From erlang@REDACTED Sun Nov 1 18:14:49 2015 From: erlang@REDACTED (Joe Armstrong) Date: Sun, 1 Nov 2015 18:14:49 +0100 Subject: [erlang-questions] Bug or feature or my ignorance? Message-ID: Is this a bug or feature or just my ignorance? I have a file which I open for write and then I do a random access pwrite at byte 11 of the file but bytes 0..10 get overwritten. Here's the example: test() -> %% make a file with 10 bytes {ok, S} = file:open("tmp",[raw,write,binary]), file:pwrite(S,0,<<0,1,2,3,4,5,6,7,8,9>>), file:close(S), %% reopen file and write from byte 11 onwards {ok, S1} = file:open("tmp",[raw,write,binary]), file:pwrite(S1,10,<<10,11,12,13,14,15>>), file:close(S1), file:read_file("tmp"). > bug:test(). {ok,<<0,0,0,0,0,0,0,0,0,0,10,11,12,13,14,15>>} If I open the file in mode [raw,write,binary,append] Then file beginning is not overwritten. So opening an existing file for writing and writing to the end of file messes up the portion of the file that I have not written to. Is this the expected behaviour? /Joe From v@REDACTED Sun Nov 1 18:17:50 2015 From: v@REDACTED (Valentin Micic) Date: Sun, 1 Nov 2015 19:17:50 +0200 Subject: [erlang-questions] Bug or feature or my ignorance? In-Reply-To: References: Message-ID: <9DA1E9B7-587A-4802-B14C-5D90E1A81078@pharos-avantgard.com> In both cases you are opening file for writing. Second time you should consider read/write in order to avoid overwriting the content. V/ On 01 Nov 2015, at 7:14 PM, Joe Armstrong wrote: > Is this a bug or feature or just my ignorance? > > I have a file which I open for write and then I do a > random access pwrite at byte 11 of the file > but bytes 0..10 get overwritten. > > Here's the example: > > > > test() -> > %% make a file with 10 bytes > {ok, S} = file:open("tmp",[raw,write,binary]), > file:pwrite(S,0,<<0,1,2,3,4,5,6,7,8,9>>), > file:close(S), > %% reopen file and write from byte 11 onwards > {ok, S1} = file:open("tmp",[raw,write,binary]), > file:pwrite(S1,10,<<10,11,12,13,14,15>>), > file:close(S1), > file:read_file("tmp"). > > >> bug:test(). > {ok,<<0,0,0,0,0,0,0,0,0,0,10,11,12,13,14,15>>} > > If I open the file in mode [raw,write,binary,append] > Then file beginning is not overwritten. > > So opening an existing file for writing and writing to the end of file > messes up the portion of the file that I have not written to. > > Is this the expected behaviour? > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sergej.jurecko@REDACTED Sun Nov 1 18:17:47 2015 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Sun, 1 Nov 2015 18:17:47 +0100 Subject: [erlang-questions] Bug or feature or my ignorance? In-Reply-To: References: Message-ID: <306800FA-6772-48C8-ABC9-CB342133717D@gmail.com> You must open with read parameter as well. Otherwise file will be truncated on open. Sergej > On 01 Nov 2015, at 18:14, Joe Armstrong wrote: > > Is this a bug or feature or just my ignorance? > > I have a file which I open for write and then I do a > random access pwrite at byte 11 of the file > but bytes 0..10 get overwritten. > > Here's the example: > > > > test() -> > %% make a file with 10 bytes > {ok, S} = file:open("tmp",[raw,write,binary]), > file:pwrite(S,0,<<0,1,2,3,4,5,6,7,8,9>>), > file:close(S), > %% reopen file and write from byte 11 onwards > {ok, S1} = file:open("tmp",[raw,write,binary]), > file:pwrite(S1,10,<<10,11,12,13,14,15>>), > file:close(S1), > file:read_file("tmp"). > > >> bug:test(). > {ok,<<0,0,0,0,0,0,0,0,0,0,10,11,12,13,14,15>>} > > If I open the file in mode [raw,write,binary,append] > Then file beginning is not overwritten. > > So opening an existing file for writing and writing to the end of file > messes up the portion of the file that I have not written to. > > Is this the expected behaviour? > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From vgarg@REDACTED Sun Nov 1 18:26:42 2015 From: vgarg@REDACTED (Vibhav Garg) Date: Sun, 1 Nov 2015 09:26:42 -0800 Subject: [erlang-questions] Erlang Benchmarks Message-ID: Are there any benchmarks for Erlang on the lines of SpecJBB for Java that can be used to compare performance between different releases as well as different types of hardware? I would love to see something that would give us a general idea of changes in performance. Thanks Vibhav -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Sun Nov 1 22:19:58 2015 From: kostis@REDACTED (Kostis Sagonas) Date: Sun, 01 Nov 2015 22:19:58 +0100 Subject: [erlang-questions] Erlang Benchmarks In-Reply-To: References: Message-ID: <563681FE.8070003@cs.ntua.gr> On 11/01/2015 06:26 PM, Vibhav Garg wrote: > Are there any benchmarks for Erlang on the lines of SpecJBB > for Java that can be used to compare > performance between different releases as well as different types of > hardware? I would love to see something that would give us a general > idea of changes in performance. Have you looked at BenchErl? http://release.softlab.ntua.gr/bencherl/ Although it does not measure just throughput as jbb2015 does, I think its infrastructure, at least, may be what you want. Comments and additions welcome. Kostis From vgarg@REDACTED Sun Nov 1 22:16:32 2015 From: vgarg@REDACTED (Vibhav Garg) Date: Sun, 1 Nov 2015 13:16:32 -0800 Subject: [erlang-questions] Erlang Benchmarks In-Reply-To: <563681FE.8070003@cs.ntua.gr> References: <563681FE.8070003@cs.ntua.gr> Message-ID: Thanks for the link! I didnt know about this benchmark suite. I will take a look Vibhav On Sun, Nov 1, 2015 at 1:19 PM, Kostis Sagonas wrote: > On 11/01/2015 06:26 PM, Vibhav Garg wrote: > >> Are there any benchmarks for Erlang on the lines of SpecJBB >> for Java that can be used to compare >> performance between different releases as well as different types of >> hardware? I would love to see something that would give us a general >> idea of changes in performance. >> > > Have you looked at BenchErl? > > http://release.softlab.ntua.gr/bencherl/ > > Although it does not measure just throughput as jbb2015 does, I think its > infrastructure, at least, may be what you want. > > Comments and additions welcome. > > Kostis > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Sun Nov 1 23:54:02 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Sun, 1 Nov 2015 23:54:02 +0100 Subject: [erlang-questions] [ANN] Sourcer: an alternative lexical scanner and parser for Erlang code Message-ID: Hello everybody! Sourcer is an Erlang scanner+parser tailored for use in tools that handle source code (like IDEs and refactoring tools), keeping track of the exact source code in the files and allowing for incomplete and malformed code. It was originally developed as part of erlide (http://erlide.org), but it is easier to handle and to get feedback if it is a separate project. As a matter of fact, I discovered many bugs, thinkos and other problems just by looking at the code as an independent entity. Source is at https://github.com/erlide/sourcer *Note:* the current status is still very alpha. I hope to get feedback from any interested party and important changes will almost certainly happen to both implementation and API. ## Rationale The normal tools for processing source code are aiming to support the compiler and thus are lossy: information that is uninteresting to the compiler is dropped and well-formedness is enforced. When supporting editors, cross-referencing tools and other tools, we find ourselves needing to access all the information about the source code and to process incomplete and ill-formed code. For example, if sections of a module are conditionally compiled and I am searching for all places where a certain function is called (maybe to rename it), I want to look even into the sections that are invisible to the regular parser because of the values of the currently defined macros. When looking at the resulting list of references, they could be annotated with the boolean macro expression deciding whether they are visible or not to the compiler. Similarly, while I am editing a file, for example starting to enter a literal string in the middle of the file, I don't want the rest of the file to get scanned as part of that string until the ending quote is entered. Having to keep an implementation of these tools parallel with the OTP ones is not easy, but the kind of functionality we are after can't be integrated into the OTP parser tools either, so we have no other option. More details are at https://github.com/erlide/sourcer/blob/master/README.md best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Nov 2 01:36:57 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 2 Nov 2015 13:36:57 +1300 Subject: [erlang-questions] pattern matching with maps in function heads In-Reply-To: <56324FEF.3000003@gmail.com> References: <56323C32.6090809@gmail.com> <56324FEF.3000003@gmail.com> Message-ID: <4CF8993E-3F46-46B4-A040-62374D698F77@cs.otago.ac.nz> On 30/10/2015, at 5:57 am, Martin Koroudjiev wrote: > Hello and thank, > > Yes, I know about the maps module but was trying to explore the map > syntax and use pattern matching. I, actually, am not trying to match on > the value but on the key name. I thought Attrib will take the value of > the first argument and this value will be used as key name in the map. For this particular example, it is 'obvious' how to schedule the steps of pattern matching to make it work. It's not so obvious in the general case. Consider f(#{X := X}) -> X. (And that's one of the easier tricky questions!) Map matching may well be generalised at some later time when the pitfalls are better understood. From ok@REDACTED Mon Nov 2 02:59:02 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 2 Nov 2015 14:59:02 +1300 Subject: [erlang-questions] New EEP 45 - FUNCTION macro In-Reply-To: References: Message-ID: <1581D0E3-EE04-43FF-8F81-935A54A2740F@cs.otago.ac.nz> On 30/10/2015, at 2:11 am, Bj?rn Gustavsson wrote: > There is a new EEP 45 that proposes a new > FUNCTION macro in the preprocessor. > > http://www.erlang.org/eeps/eep-0045.html > https://github.com/erlang/eep/blob/master/eeps/eep-0045.md The problem with ?FUNCTION => {Name,Arity} is that obviously sensible things like fun ?FUNCTION don't work. The problem with ?FUNCTION => Name/Arity would be that obviously sensible things like Name/_Arity = ?FUNCTION wouldn't work. Why not cut the Gordian knot? A __FUNCTION__ macro for C makes sense because the arity of a C function is not part of its name (but part of its type). For Erlang, I respectfully suggest that it makes more sense to have *two* macros: ?FUNCTION_NAME => the name of the containing function ?FUNCTION_ARITY => the arity of the containing function and then people can easily write whatever they want. For example, f(G, X) when is_function(G, ?FUNCTION_ARITY) -> ... is a lot clearer than anything involving ?FUNCTION. The really good thing about defining ?FUNCTION_NAME and ?FUNCTION_ARITY is that anyone who *wants* ?FUNCTION is then at liberty to -define(?FUNCTION, {?FUNCTION_NAME,?FUNCTION_ARITY}). if that's what they need. I did notice that there are definitions of ?FUNCTION_NAME and ?FUNCTION_ARITY in the EEP *** BUT THEY DOEs NOT WORK IN FUNCTION HEADS ** Nor for that matter will they work after 'fun'. This makes the claim that ?FUNCTION_NAME and ?FUNCTION_ARITY can "easily be defined in terms of FUNCTION" false. Given the asymmetry, it is obvious which should be taken as basic and which as derived. Looking at the text, "must be efficient ... slowdown" appears to mean macro-expansion time. It would be good to be explicit about that. As for FUNCTION_STRING, it's not clear why it's -define(FUNCTION_STRING, atom_to_list(?FUNCTION_NAME) ++ "/" ++ integer_to_list(?FUNCTION_ARITY)). rather than -define(FUNCTION_STRING,\ atom_to_list(?MODULE) ++ ":" ++ atom_to_list(?FUNCTION_NAME) ++ "/" ++ integer_to_list(?FUNCTION_ARITY)). From ok@REDACTED Mon Nov 2 04:13:38 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 2 Nov 2015 16:13:38 +1300 Subject: [erlang-questions] Fourth draft of EEP 44 - Additional preprocessor directives In-Reply-To: References: Message-ID: <8018932F-D641-42F5-BB97-2F7CF7D017B4@cs.otago.ac.nz> I am still unhappy that we cannot conditionally select *clauses*. I do know that it is possible to do -if... -define(WANTED, true). -else. -define(WANTED, false). -endif. ... f(...) f(...) when ?WANTED f(...) *But* as EEP 44 reminds us, "There is one thing a preprocessor, and only a preprocessor can do: skip code that is not syntactically correct (for example, code that uses the map syntax)." In fact this proposal *still* does not give us a way to skip one or more clauses that *would* be syntactically incorrect in some context. There is a very simple change to Erlang that would make this possible: allow '.' between function clauses as well as ';'. That would also enable us to paste a function together out of pieces held in separate files: % Define f/3 -include("old_f.hrl"). -include("new_f.hrl"). That aside, I think this draft is pretty good. I don't quite buy the argument against 'supported': Perhaps. It seems that for this to work the list of supported feature names accepted by supported must be carefully maintained and documented for each release. The users must then look up the appropriate feature name to use. It may also not be obvious how to name minor changes to the type specification syntax or to the language itself. The resulting code may not be any easier to understand than a test against a release number. Considering the needs of the *writer*, ?OTP_RELEASE is easier than 'supported'. The list of features supported by each release "must be carefully maintained and documented for each release" *anyway*. But considering the needs of the *maintainer*, if I see just -if(OTP_RELEASE > 20). then I have no clue about *why* that test is there. I have to search the documentation (and let us hope tha/*fenv there _is_ a documentation file listing all and only the things that you might use OTP_RELEASE for) and then guess which of perhaps ten reasons is the correct one. Amongst other things, since features do sometimes go away, I am left with no clue about whether I ought to replace that by -if(OTP_RELEASE > 20 andalso OTP_RELEASE < 25). I think I can confidently say that an -if using names *will* be easier to understand than one using OTP_RELEASE. I do accept that 'supported' is something that can well be added in a later EEP, and I would not take the clunkiness of OTP_RELEASE as a show-stopper. But the documentation should insist that anyone using OTP_RELEASE write an explicit comment clearly stating what they are really testing for. Let me give you an analogy. In the support code for my Smalltalk system, there are tests for __APPLE__, __OpenBSD__, __linux, and __Solaris. I'll spare you the details, but - common functions in different headers or even lacking headers - common features requiring DIFFERENT feature-test macros to get them - no better way to discover what you need to do than testing which system it is. As a result, #ifdef __APPLE__ could really be asking at least four different questions. It's usually pretty clear which, but I see I'm going to have to improve my internal documentation. Yes, I know that this is exactly the kind of thing that configure and friends exist to solve, but if you're trying to keep GNU stuff out of your program, what can you do? I'm saying that using OTP_RELEASE is just as confusing as using __APPLE__. There is a way to test if a *header* exists (is_header/1), which looks in the header search path. There is a way to test if a (compiled) *module* exists (is_module/1), which looks in the module search path. I am convinced that these are good ideas. If only C had them! But there is no way to test if a *data file* exists. From bchesneau@REDACTED Mon Nov 2 16:53:36 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 02 Nov 2015 15:53:36 +0000 Subject: [erlang-questions] how to check if an application has been upgraded Message-ID: I am for now polling from time to time the applications I have registered to watch if one have been upgraded and then optionally discover some resources from it. But I am wondering if there is a bette way to know if an application have been upgraded, or restarted from another application. Or rather what would be the more efficient way to do it? - beno?t -------------- next part -------------- An HTML attachment was scrubbed... URL: From kvratkin@REDACTED Mon Nov 2 17:59:18 2015 From: kvratkin@REDACTED (Kirill Ratkin) Date: Mon, 2 Nov 2015 19:59:18 +0300 Subject: [erlang-questions] Stop C-port in case of erlang process killed Message-ID: Hi, I'm developing C-based program which should work as port. Issue is when erlang process hardly killed (by exit(Pid,kill)) from observer for example. In this case C-process doesn't see what happen in Erlang node and continues to work. Is there some way to catch such situation in C? It seems C-process sees nothing when erlang processe killed. I tried to catch signals (SIGPIPE and others). I run C-process under 'strace' and tried to gdb it. There are no messages/signals come to C-process excepting some read/write periodical messages which looks like heartbeat. Maybe I need to make some 'monitor' or 'link' of Erlang process from C-process ... BR Kirill -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.santos@REDACTED Mon Nov 2 18:36:56 2015 From: michael.santos@REDACTED (Michael Santos) Date: Mon, 2 Nov 2015 12:36:56 -0500 Subject: [erlang-questions] Stop C-port in case of erlang process killed In-Reply-To: References: Message-ID: <20151102173656.GA13886@brk> On Mon, Nov 02, 2015 at 07:59:18PM +0300, Kirill Ratkin wrote: > Hi, > > I'm developing C-based program which should work as port. > Issue is when erlang process hardly killed (by exit(Pid,kill)) from > observer for example. In this case C-process doesn't see what happen in > Erlang node and continues to work. Is there some way to catch such > situation in C? In the port, check for EOF when doing the read on stdin and exit. A port is basically written like a component in a shell pipeline: 1> P = open_port({spawn, "/bin/cat"}, [binary]). #Port<0.749> 2> erlang:port_info(P). [{name,"/bin/cat"}, {links,[<0.52.0>]}, {id,11984}, {connected,<0.52.0>}, {input,0}, {output,0}, {os_pid,5465}] 3> erlang:port_command(P, "foobar\n"). true 4> flush(). Shell got {#Port<0.749>,{data,<<"foobar\n">>}} ok 4> self(). <0.61.0> % shell process owns the port: killing the Erlang process kills the port 5> a = b. ** exception error: no match of right hand side value b 6> self(). <0.64.0> 7> erlang:port_command(P, "foobar\n"). ** exception error: bad argument in function port_command/2 called as port_command(#Port<0.749>,"foobar\n") > It seems C-process sees nothing when erlang processe killed. I tried to > catch signals (SIGPIPE and others). The port will receive a SIGPIPE if it tries to write to stdout. > I run C-process under 'strace' and > tried to gdb it. There are no messages/signals come to C-process excepting > some read/write periodical messages which looks like heartbeat. > > Maybe I need to make some 'monitor' or 'link' of Erlang process from > C-process ... > > BR > Kirill > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From charles@REDACTED Mon Nov 2 20:50:08 2015 From: charles@REDACTED (Charles Weitzer) Date: Mon, 2 Nov 2015 19:50:08 +0000 Subject: [erlang-questions] Senior Software Engineer Needed for Machine Learning Startup Message-ID: Voleon Capital Management LP is a startup quantitative hedge fund located in Berkeley, California. We would like to hire a senior software engineer as soon as possible. Voleon's founders previously worked together at one of the most successful quantitative hedge funds in the world. Our CEO has a PhD in Computer Science from Stanford and has been CEO and founder of a successful Internet infrastructure startup. Our Chief Investment Officer has a PhD in Statistics from Berkeley. Voleon's team includes PhD's from leading departments in statistics, computer science, and mathematics. We have made several unpublished advances in the field of machine learning and in other areas as well. Here is our formal job description: ********************************************************** * Senior Software Engineer * Technology-driven investment firm employing cutting-edge statistical machine learning techniques seeks an exceptionally capable software engineer. You will architect and implement new production trading systems, machine learning infrastructure, data integration pipelines, and large-scale storage systems. The firm researches and deploys systematic trading strategies designed to generate attractive returns without being dependent on the performance of the overall market. Join a team of under 30 people that includes a Berkeley statistics professor as well as over ten PhD's from Berkeley, Chicago, CMU, Princeton, Stanford, and UCLA, led by the founder and CEO of a successful Internet infrastructure technology firm. The firm's offices are walking distance from BART and the UC Berkeley campus in downtown Berkeley, California. We have a casual and collegial office environment, weekly catered lunches, and competitive benefits packages. We seek candidates with a proven track record of writing correct, well-designed software, solving hard problems, and delivering complex projects on time. You should preferably have experience designing and implementing fault-tolerant distributed systems. Experience with building large-scale data infrastructure, stream processing systems, or latency-sensitive programs is a bonus. We are growing rapidly. Willingness to take initiative and a gritty determination to productize are essential. Required experience: - experience with functional programming languages such as Erlang, Haskell, etc. - developing with C/C++/Python/Go in a Linux environment with a focus on performance, concurrency, and correctness. - working in TCP/IP networking, multi-threading, and server development. - working with common Internet protocols (IP, TCP/UDP, SSL/TLS, HTTP, SNMP, etc.). - architecting and designing highly available systems. - architecting and designing large-scale data management infrastructure. - working in large codebases and building modular, manageable code. Preferred experience: - debugging and performance profiling, including the use of tools such as strace, valgrind, gdb, tcpdump, etc. - working with build and test automation tools. - working with well-defined change management processes. - diagnosing RDBMS performance problems, exploiting indexing, using EXPLAIN PLAN, optimizing at the code layer, etc. - working with messaging queues (RabbitMQ, Redis, etc.) as well as distributed caching systems. Interest in financial applications is essential, but experience in finance is not a primary factor in our hiring. Benefits and compensation are highly competitive. ********************************************************** The above job description is just a starting point in terms of possible duties and seniority. We can be very flexible for the right person. If you are interested, please apply with your full and complete CV: http://voleon.com/apply/ The Voleon Group is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco.molteni@REDACTED Mon Nov 2 22:40:01 2015 From: marco.molteni@REDACTED (Marco Molteni) Date: Mon, 2 Nov 2015 22:40:01 +0100 Subject: [erlang-questions] erl_call + eunit: how to redirect eunit output to erl_call? Message-ID: <348F30BE-A2BA-4262-9A3D-CB9AB05E1A97@laposte.net> Hi all, to be able to do real test-driven development, one needs extremely fast unit tests (ideally less than 10ms per test case). To run the unit tests, one needs the Erlang VM, which on my laptop has an overhead of around 1.2 - 1.5 secs: $ time erl -noshell -s init stop real 0m1.217s (clearly for a long-running Erlang system this is not a problem at all :-) If we run an empty UT, we get: $ time erl -pa bin -noshell -eval 'eunit:test("bin", [verbose])' -s init stop ======================== EUnit ======================== directory "bin" ftp30_test: simple_test (module 'ftp30_test')...ok [done in 0.009 s] ======================================================= Test passed. real 0m1.250s confirming the overhead. I was thinking: is it possible to shave off this overhead, when running the UTs? I found out about erl_call [1] and I was excited, so in one terminal I started the VM as: erl -pa ebin -sname dust and in another terminal I ran: $ time erl_call -sname dust -a 'eunit test ["ebin", [verbose]]' ok real 0m0.024s Now, if you look at the time, it took 24 msec as opposed to the traditional 1250 msec, so it looks good, but, there is no eunit output! The classic eunit output still goes to the first terminal, where I launched the VM: ======================== EUnit ======================== directory "ebin" ftp30_test: simple_test (module 'ftp30_test')...ok [done in 0.011 s] ======================================================= Test passed. I know that eunit plays tricks with stdout, I was wondering if there was a way to redirect the output so that one can get it out of erl_call. I am willing to attempt to make the changes, but I'd like some guidance :-) And, I also realize that I am not counting the compile time and that when the number of tests becomes big enough, the 1.2 sec overhead will be negligible. On the other hand, I am used to blazingly fast C++ TDD (when not compiling templated code ;-), and I think that to do TDD one needs extremely fast feedback. Even waiting 1 sec doesn't feel right :-) Or maybe another path to consider is not using erl_call at all and just using the VM, with a stripped-down environment like the SEE explained in Joe's second edition Programming Erlang... thanks marco.m [1] http://www.erlang.org/doc/man/erl_call.html From eric.pailleau@REDACTED Mon Nov 2 23:44:20 2015 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Mon, 2 Nov 2015 23:44:20 +0100 Subject: [erlang-questions] how to check if an application has been upgraded In-Reply-To: References: Message-ID: <5637E744.8020200@wanadoo.fr> Le 02/11/2015 16:53, Benoit Chesneau a ?crit : > I am for now polling from time to time the applications I have > registered to watch if one have been upgraded and then optionally > discover some resources from it. But I am wondering if there is a bette > way to know if an application have been upgraded, or restarted from > another application. > > Or rather what would be the more efficient way to do it? Hi Benoit, this thread may help ? http://erlang.org/pipermail/erlang-questions/2015-June/084873.html regards From mmartin4242@REDACTED Tue Nov 3 02:21:14 2015 From: mmartin4242@REDACTED (Michael Martin) Date: Mon, 2 Nov 2015 19:21:14 -0600 Subject: [erlang-questions] eunit question Message-ID: <56380C0A.6050408@gmail.com> Hi all, I'm a bit perplexed about this one. eunit is giving me an "unexpected success" when testing a function with guards. The source code and the output from ./rebar eu are attached. Any insight would be greatly appreciated. Thanks, Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: test1.erl Type: text/x-erlang Size: 616 bytes Desc: not available URL: -------------- next part -------------- ==> test1 (eunit) ======================== EUnit ======================== module 'test1_sup' test1: test1_test (module 'test1')...src/test1.erl:11:<0.46.0>: Argument: not_a_binary src/test1.erl:12:<0.46.0>: Is a binary: false *failed* in function test1:'-test1_test/0-fun-0-'/1 (src/test1.erl, line 22) **error:{assertException_failed,[{module,test1}, {line,22}, {expression,"test1 ( TestMap , not_a_binary )"}, {pattern,"{ error , function_clause , [...] }"}, {unexpected_success,ok}]} module 'test1_app' [done in 0.001 s] ======================================================= Failed: 1. Skipped: 0. Passed: 0. Cover analysis: /tmp/test1/.eunit/index.html Code Coverage: test1 : 100% test1_app : 0% test1_sup : 0% Total : 55% ERROR: One or more eunit tests failed. ERROR: eunit failed while processing /tmp/test1: rebar_abort From kostis@REDACTED Tue Nov 3 02:33:04 2015 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 03 Nov 2015 02:33:04 +0100 Subject: [erlang-questions] eunit question In-Reply-To: <56380C0A.6050408@gmail.com> References: <56380C0A.6050408@gmail.com> Message-ID: <56380ED0.2000009@cs.ntua.gr> On 11/03/2015 02:21 AM, Michael Martin wrote: > > I'm a bit perplexed about this one. eunit is giving me an "unexpected > success" when testing a function with guards. > The source code and the output from ./rebar eu are attached. > Any insight would be greatly appreciated. > > -spec test1(map(), binary()) -> map(). > > test1(Data, Argument) when is_map(Data); is_binary(Argument) -> You most likely want to put a ',' instead of a ';' in the guard. Kostis From mmartin4242@REDACTED Tue Nov 3 02:27:26 2015 From: mmartin4242@REDACTED (Michael Martin) Date: Mon, 2 Nov 2015 19:27:26 -0600 Subject: [erlang-questions] eunit question In-Reply-To: <56380ED0.2000009@cs.ntua.gr> References: <56380C0A.6050408@gmail.com> <56380ED0.2000009@cs.ntua.gr> Message-ID: <56380D7E.5060808@gmail.com> Thanks, Kostis, that fixed it. Very strange, though, as all my other functions have a ';' rather than a ',', and they all work. I'm changing them all to commas now. Thanks again! On 11/02/2015 07:33 PM, Kostis Sagonas wrote: > On 11/03/2015 02:21 AM, Michael Martin wrote: >> >> I'm a bit perplexed about this one. eunit is giving me an "unexpected >> success" when testing a function with guards. >> The source code and the output from ./rebar eu are attached. >> Any insight would be greatly appreciated. >> >> -spec test1(map(), binary()) -> map(). >> >> test1(Data, Argument) when is_map(Data); is_binary(Argument) -> > > You most likely want to put a ',' instead of a ';' in the guard. > > Kostis From g.a.vinogradov@REDACTED Tue Nov 3 03:31:09 2015 From: g.a.vinogradov@REDACTED (Gleb Vinogradov) Date: Tue, 3 Nov 2015 08:31:09 +0600 Subject: [erlang-questions] eunit question In-Reply-To: <56380D7E.5060808@gmail.com> References: <56380C0A.6050408@gmail.com> <56380ED0.2000009@cs.ntua.gr> <56380D7E.5060808@gmail.com> Message-ID: Hi, Michael No-no-no, you shouldn't change them all to commas! They are different - comma act as AND, while semicolon act as OR. Good luck! Gleb 2015-11-03 7:27 GMT+06:00 Michael Martin : > Thanks, Kostis, that fixed it. > Very strange, though, as all my other functions have a ';' rather than a > ',', and they all work. > I'm changing them all to commas now. > > Thanks again! > > > > On 11/02/2015 07:33 PM, Kostis Sagonas wrote: > >> On 11/03/2015 02:21 AM, Michael Martin wrote: >> >>> >>> I'm a bit perplexed about this one. eunit is giving me an "unexpected >>> success" when testing a function with guards. >>> The source code and the output from ./rebar eu are attached. >>> Any insight would be greatly appreciated. >>> >>> -spec test1(map(), binary()) -> map(). >>> >>> test1(Data, Argument) when is_map(Data); is_binary(Argument) -> >>> >> >> You most likely want to put a ',' instead of a ';' in the guard. >> >> Kostis >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Tue Nov 3 07:51:32 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Mon, 2 Nov 2015 22:51:32 -0800 Subject: [erlang-questions] eunit question In-Reply-To: References: <56380C0A.6050408@gmail.com> <56380ED0.2000009@cs.ntua.gr> <56380D7E.5060808@gmail.com> Message-ID: <56385974.1060900@gmail.com> On 11/02/2015 06:31 PM, Gleb Vinogradov wrote: > Hi, Michael > > No-no-no, you shouldn't change them all to commas! > They are different - comma act as AND, while semicolon act as OR. Indeed! I suspect that these links might be useful: http://erlang.org/doc/reference_manual/expressions.html#id83606 http://erlang.org/doc/reference_manual/functions.html#syntax -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From cbenac@REDACTED Tue Nov 3 11:14:18 2015 From: cbenac@REDACTED (Clara Benac Earle) Date: Tue, 03 Nov 2015 11:14:18 +0100 Subject: [erlang-questions] [ANN] Madrid Erlounge Nov. 4th 19:00ish :-) Message-ID: <563888FA.1060609@fi.upm.es> Dear all, Apologies for the late announcement. Tomorrow Nov. 4th at 19:00 we will meet for a Madrid Erlounge at the usual place (Sala de Grados (1? planta Facultad de Inform?tica, Universidad Complutense de Madrid) (http://gpd.sip.ucm.es/fraguas/wflp06/campus_ucm.jpg). Manuel Rubio and Guillermo Rodr?guez from http://altenwald.org/ will tell us how they use the Erlang magic in their projects and why they manage to get ahead of their competitors thanks to Erlang. If you want to be informed of Madrid Erlang events in advance, please join the madrid-erlang-users mailing list https://babel.ls.fi.upm.es/cgi-bin/mailman/listinfo/madrid-erlang-users See you there! Clara From jesper.louis.andersen@REDACTED Tue Nov 3 12:50:33 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 3 Nov 2015 12:50:33 +0100 Subject: [erlang-questions] erl_call + eunit: how to redirect eunit output to erl_call? In-Reply-To: <348F30BE-A2BA-4262-9A3D-CB9AB05E1A97@laposte.net> References: <348F30BE-A2BA-4262-9A3D-CB9AB05E1A97@laposte.net> Message-ID: On Mon, Nov 2, 2015 at 10:40 PM, Marco Molteni wrote: > Or maybe another path to consider is not using erl_call at all and just > using the VM, with a stripped-down environment like the SEE explained in > Joe's second edition Programming Erlang... > I'd look at some tool in the vein of Rusty's sync: https://github.com/rustyio/sync and add a hook to it. That way, you just run a server continually and load code into it on the fly as you are building it. You don't even have to prod it with something like erl_call if this is what you do. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmartin4242@REDACTED Tue Nov 3 14:47:34 2015 From: mmartin4242@REDACTED (Michael Martin) Date: Tue, 3 Nov 2015 07:47:34 -0600 Subject: [erlang-questions] eunit question In-Reply-To: <56385974.1060900@gmail.com> References: <56380C0A.6050408@gmail.com> <56380ED0.2000009@cs.ntua.gr> <56380D7E.5060808@gmail.com> <56385974.1060900@gmail.com> Message-ID: <5638BAF6.8070905@gmail.com> Thanks everyone. In my case, commas are appropriate, semi-colons not. It's all working as expected now. Thanks again! On 11/03/2015 12:51 AM, Kenneth Lakin wrote: > On 11/02/2015 06:31 PM, Gleb Vinogradov wrote: >> Hi, Michael >> >> No-no-no, you shouldn't change them all to commas! >> They are different - comma act as AND, while semicolon act as OR. > Indeed! I suspect that these links might be useful: > > http://erlang.org/doc/reference_manual/expressions.html#id83606 > http://erlang.org/doc/reference_manual/functions.html#syntax > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Tue Nov 3 15:52:26 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 03 Nov 2015 14:52:26 +0000 Subject: [erlang-questions] how to check if an application has been upgraded In-Reply-To: <5637E744.8020200@wanadoo.fr> References: <5637E744.8020200@wanadoo.fr> Message-ID: On Mon, Nov 2, 2015 at 11:44 PM PAILLEAU Eric wrote: > Le 02/11/2015 16:53, Benoit Chesneau a ?crit : > > I am for now polling from time to time the applications I have > > registered to watch if one have been upgraded and then optionally > > discover some resources from it. But I am wondering if there is a bette > > way to know if an application have been upgraded, or restarted from > > another application. > > > > Or rather what would be the more efficient way to do it? > Hi Benoit, > this thread may help ? > > http://erlang.org/pipermail/erlang-questions/2015-June/084873.html > > regards > > > Thanks for the link! Looks like it could work for my case. But doesn't it prevent the usage of dbg later? Or any pro-active tracing while running? - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Tue Nov 3 16:41:07 2015 From: t@REDACTED (Tristan Sloughter) Date: Tue, 03 Nov 2015 09:41:07 -0600 Subject: [erlang-questions] erl_call + eunit: how to redirect eunit output to erl_call? In-Reply-To: References: <348F30BE-A2BA-4262-9A3D-CB9AB05E1A97@laposte.net> Message-ID: <1446565267.3321079.427909017.255F9B45@webmail.messagingengine.com> Or http://blog.erlware.org/rebar3-auto-comile-and-load-plugin/ Eventually I plan to have `auto` support running eunit tests of only modules that have changed (or a module that the eunit test depends on changes), so you can simply run `rebar3 auto eunit` and when you save changes it'll automatically only run the necessary tests to verify your changes. -- Tristan Sloughter t@REDACTED On Tue, Nov 3, 2015, at 05:50 AM, Jesper Louis Andersen wrote: > > On Mon, Nov 2, 2015 at 10:40 PM, Marco Molteni wrote: >> Or maybe another path to consider is not using erl_call at all and just using the VM, with a stripped-down environment like the SEE explained in Joe's second edition Programming Erlang... > I'd look at some tool in the vein of Rusty's sync: https://github.com/rustyio/sync and add a hook to it. That way, you just run a server continually and load code into it on the fly as you are building it. You don't even have to prod it with something like erl_call if this is what you do. > > > -- > J. > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianfranco.alongi@REDACTED Tue Nov 3 17:15:48 2015 From: gianfranco.alongi@REDACTED (Gianfranco Alongi) Date: Tue, 3 Nov 2015 17:15:48 +0100 Subject: [erlang-questions] erl_call + eunit: how to redirect eunit output to erl_call? In-Reply-To: <1446565267.3321079.427909017.255F9B45@webmail.messagingengine.com> References: <348F30BE-A2BA-4262-9A3D-CB9AB05E1A97@laposte.net> <1446565267.3321079.427909017.255F9B45@webmail.messagingengine.com> Message-ID: This seems similar to Active-Ring, a 5 year old project. Look into it - there might be something inspirational there. https://github.com/domiwilliams/active-ring On Tue, Nov 3, 2015 at 4:41 PM, Tristan Sloughter wrote: > Or http://blog.erlware.org/rebar3-auto-comile-and-load-plugin/ > > Eventually I plan to have `auto` support running eunit tests of only > modules that have changed (or a module that the eunit test depends on > changes), so you can simply run `rebar3 auto eunit` and when you save > changes it'll automatically only run the necessary tests to verify your > changes. > > -- > Tristan Sloughter > t@REDACTED > > > > On Tue, Nov 3, 2015, at 05:50 AM, Jesper Louis Andersen wrote: > > > On Mon, Nov 2, 2015 at 10:40 PM, Marco Molteni > wrote: > > Or maybe another path to consider is not using erl_call at all and just > using the VM, with a stripped-down environment like the SEE explained in > Joe's second edition Programming Erlang... > > I'd look at some tool in the vein of Rusty's sync: > https://github.com/rustyio/sync and add a hook to it. That way, you just > run a server continually and load code into it on the fly as you are > building it. You don't even have to prod it with something like erl_call if > this is what you do. > > > -- > J. > *_______________________________________________* > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Tue Nov 3 18:21:26 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Tue, 03 Nov 2015 18:21:26 +0100 Subject: [erlang-questions] how to check if an application has been upgraded In-Reply-To: Message-ID: <1ac84775-6ffc-4862-ab38-2f2a76be2729@email.android.com> An HTML attachment was scrubbed... URL: From jay@REDACTED Tue Nov 3 18:25:04 2015 From: jay@REDACTED (Jay Nelson) Date: Tue, 3 Nov 2015 09:25:04 -0800 Subject: [erlang-questions] how to check if an application has been upgraded Message-ID: <3F5A10BD-0DF0-486C-9F0C-7A483DA41815@duomark.com> > I am for now polling from time to time the applications I have registered > to watch if one have been upgraded and then optionally discover some > resources from it. Presuming you are using standard relup/reldown you should be able to add a module to each application which will receive the upgrade / downgrade notifications. Just make it an fsm that can transition on appropriate appup instructions and then continue running in the upgraded state until you?ve discovered the resources, after which it can transition back to the ?WAITING_FOR_UPGRADE? state. jay From mjtruog@REDACTED Tue Nov 3 20:53:32 2015 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 03 Nov 2015 11:53:32 -0800 Subject: [erlang-questions] New EEP 45 - FUNCTION macro In-Reply-To: <1581D0E3-EE04-43FF-8F81-935A54A2740F@cs.otago.ac.nz> References: <1581D0E3-EE04-43FF-8F81-935A54A2740F@cs.otago.ac.nz> Message-ID: <563910BC.4060200@gmail.com> On 11/01/2015 05:59 PM, Richard A. O'Keefe wrote: > On 30/10/2015, at 2:11 am, Bj?rn Gustavsson wrote: > >> There is a new EEP 45 that proposes a new >> FUNCTION macro in the preprocessor. >> >> http://www.erlang.org/eeps/eep-0045.html >> https://github.com/erlang/eep/blob/master/eeps/eep-0045.md > The problem with ?FUNCTION => {Name,Arity} is that > obviously sensible things like > fun ?FUNCTION > don't work. > > The problem with ?FUNCTION => Name/Arity would be that > obviously sensible things like > Name/_Arity = ?FUNCTION > wouldn't work. > > Why not cut the Gordian knot? > A __FUNCTION__ macro for C makes sense because the arity of > a C function is not part of its name (but part of its type). > For Erlang, I respectfully suggest that it makes more sense > to have *two* macros: > > ?FUNCTION_NAME => the name of the containing function > ?FUNCTION_ARITY => the arity of the containing function > > and then people can easily write whatever they want. For example, > > f(G, X) when is_function(G, ?FUNCTION_ARITY) -> ... > > is a lot clearer than anything involving ?FUNCTION. > > The really good thing about defining ?FUNCTION_NAME and > ?FUNCTION_ARITY is that anyone who *wants* ?FUNCTION is > then at liberty to > -define(?FUNCTION, {?FUNCTION_NAME,?FUNCTION_ARITY}). > if that's what they need. > > I did notice that there are definitions of ?FUNCTION_NAME > and ?FUNCTION_ARITY in the EEP > > *** BUT THEY DOEs NOT WORK IN FUNCTION HEADS ** > > Nor for that matter will they work after 'fun'. > > This makes the claim that ?FUNCTION_NAME and ?FUNCTION_ARITY > can "easily be defined in terms of FUNCTION" false. > Given the asymmetry, it is obvious which should be taken as > basic and which as derived. > > Looking at the text, > "must be efficient ... slowdown" > appears to mean macro-expansion time. It would be good to be > explicit about that. > > As for FUNCTION_STRING, it's not clear why it's > -define(FUNCTION_STRING, atom_to_list(?FUNCTION_NAME) ++ "/" ++ > integer_to_list(?FUNCTION_ARITY)). > rather than > -define(FUNCTION_STRING,\ > atom_to_list(?MODULE) ++ ":" ++ > atom_to_list(?FUNCTION_NAME) ++ "/" ++ > integer_to_list(?FUNCTION_ARITY)). > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > Thank you for adding EEP 45! I agree that the feature is the most flexible in Erlang when using the two separate macros: ?FUNCTION_NAME and ?FUNCTION_ARITY . Using the macros to create the function as data is a good use-case: F = fun ?FUNCTION_NAME/?FUNCTION_ARITY, ... which is similar to using the ?MODULE macro to avoid problems with module name changes during development. From mmartin4242@REDACTED Tue Nov 3 22:35:17 2015 From: mmartin4242@REDACTED (Michael Martin) Date: Tue, 3 Nov 2015 15:35:17 -0600 Subject: [erlang-questions] meck:validate failure Message-ID: <56392895.1080609@gmail.com> Hi all, I have a couple of meck:validates that are failing, and there's really no diagnostic help from this call that I can see. Googling hasn't helped, either. What's the best approach to debugging these kinds of failures? Thanks, Michael From dodul@REDACTED Tue Nov 3 23:05:51 2015 From: dodul@REDACTED (Kamrul Khan) Date: Wed, 4 Nov 2015 04:05:51 +0600 Subject: [erlang-questions] How to build and use epgsql (erlang) Message-ID: My erlnag program directory structure is like the below:src ?pgtest.erl lib ?epgsql ?rebar I downloaded the epgsql library from https://github.com/epgsql/epgsql then tried to build it with make which gave me the below error:make: rebar: Command not found make: *** [compile] Error 127So then I downloaded rebar from https://github.com/basho/rebar and build it. It was build successfully and gave me the message. Congratulations! You now have a self-contained script called ?rebar? in your current working directory. Place this script anywhere in your path and you can use rebar to build OTP-compliant apps. So as per my understanding I copied the rebar script from rebar directory to my epgsql directory and then tried to make again. However, I got the same error. I know I am doing it wrong, but not sure what exactly to do? -------------- next part -------------- An HTML attachment was scrubbed... URL: From povetkin.k@REDACTED Tue Nov 3 23:38:16 2015 From: povetkin.k@REDACTED (Constantine Povietkin) Date: Wed, 4 Nov 2015 00:38:16 +0200 Subject: [erlang-questions] Test if function has been called Message-ID: <20151104003816.06c78538@kappac.home.local> Hi all, I know my question has been discussed _a lot of_ times, but I've no luck to find an answer. I've a gen_server based module that should call handlers with apply. Method that calls handlers should not return anything by design. What is(are) way(s) to test if function has been called. Thanks in advance! P.S. Links to tools/documentation are highly appreciated. From eiri@REDACTED Tue Nov 3 23:59:50 2015 From: eiri@REDACTED (Eiri) Date: Tue, 3 Nov 2015 18:59:50 -0400 Subject: [erlang-questions] meck:validate failure In-Reply-To: <56392895.1080609@gmail.com> References: <56392895.1080609@gmail.com> Message-ID: meck:history/1 returns a list of all the calls to the mocked module, including the unexpected ones. Regards, Eric > On Nov 3, 2015, at 5:35 PM, Michael Martin wrote: > > Hi all, > > I have a couple of meck:validates that are failing, and there's really no diagnostic help from > this call that I can see. Googling hasn't helped, either. What's the best approach to debugging > these kinds of failures? > > Thanks, > Michael > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From kennethlakin@REDACTED Wed Nov 4 09:40:42 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 4 Nov 2015 00:40:42 -0800 Subject: [erlang-questions] How to build and use epgsql (erlang) In-Reply-To: References: Message-ID: <5639C48A.1060907@gmail.com> On 11/03/2015 02:05 PM, Kamrul Khan wrote: > So as per my understanding I copied the rebar script from rebar > directory to my epgsql directory and then tried to make again. However, > I got the same error. I know I am doing it wrong, but not sure what > exactly to do? You want to put the executable named "rebar" created by building the rebar project somewhere in your PATH. In a terminal, type echo $PATH to see where you all the places where you could -currently- put rebar. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From paulperegud@REDACTED Wed Nov 4 09:47:03 2015 From: paulperegud@REDACTED (Paul Peregud) Date: Wed, 4 Nov 2015 09:47:03 +0100 Subject: [erlang-questions] Test if function has been called In-Reply-To: <20151104003816.06c78538@kappac.home.local> References: <20151104003816.06c78538@kappac.home.local> Message-ID: You can use tracing. Redbug from eper is pretty easy to use. Documentation is in redbug.erl file. https://github.com/massemanet/eper On Tue, Nov 3, 2015 at 11:38 PM, Constantine Povietkin wrote: > Hi all, > > I know my question has been discussed _a lot of_ times, but I've no > luck to find an answer. > > I've a gen_server based module that should call handlers with apply. > Method that calls handlers should not return anything by design. > > What is(are) way(s) to test if function has been called. > > Thanks in advance! > > P.S. Links to tools/documentation are highly appreciated. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Paul Peregud +48602112091 From bchesneau@REDACTED Wed Nov 4 17:00:31 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Wed, 04 Nov 2015 16:00:31 +0000 Subject: [erlang-questions] merl : loading vs storing as beam Message-ID: Hi, for an application i was using merl to create dynamically a module. I am for now just loading it in memory using `merl:compile_and_load/2` . But I am wondering if I should also store it on the file system. Is there any reason to do it on a live system since the module will always be regenerated on restart? - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Wed Nov 4 21:54:16 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 4 Nov 2015 21:54:16 +0100 Subject: [erlang-questions] merl : loading vs storing as beam In-Reply-To: References: Message-ID: Not really, unless you want to cache it for faster startup, if it's a very large generated module (or several). /Richard On Wed, Nov 4, 2015 at 5:00 PM, Benoit Chesneau wrote: > Hi, > > for an application i was using merl to create dynamically a module. I am > for now just loading it in memory using `merl:compile_and_load/2` . > > But I am wondering if I should also store it on the file system. Is there > any reason to do it on a live system since the module will always be > regenerated on restart? > > - benoit > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Nov 5 10:14:37 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 05 Nov 2015 09:14:37 +0000 Subject: [erlang-questions] merl : loading vs storing as beam In-Reply-To: References: Message-ID: thanks for the answer! will let it in memory then On Wed, 4 Nov 2015 at 21:54, Richard Carlsson wrote: > Not really, unless you want to cache it for faster startup, if it's a very > large generated module (or several). > > > /Richard > > On Wed, Nov 4, 2015 at 5:00 PM, Benoit Chesneau > wrote: > >> Hi, >> >> for an application i was using merl to create dynamically a module. I am >> for now just loading it in memory using `merl:compile_and_load/2` . >> >> But I am wondering if I should also store it on the file system. Is there >> any reason to do it on a live system since the module will always be >> regenerated on restart? >> >> - benoit >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Thu Nov 5 14:26:06 2015 From: bog495@REDACTED (Bogdan Andu) Date: Thu, 5 Nov 2015 15:26:06 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt Message-ID: Hi, I have a stream of data called EncryptedData (with randomiv header specification). I want to decrypt using: crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). where: EncryptedData is formatted as follows: <> IvSpec == <<"RandomIV">> However the decryption does not works and garbage is generated (the function ) . What could be done given the fact that IV is an 8-byte quantity the Key is formatted according to crypto test suites and is 16 bytes long Am I missing something? Thanks, Bogdan -------------- next part -------------- An HTML attachment was scrubbed... URL: From radek@REDACTED Thu Nov 5 14:38:13 2015 From: radek@REDACTED (Radoslaw Gruchalski) Date: Thu, 5 Nov 2015 13:38:13 +0000 (UTC) Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: Message-ID: You say your data is: <> == <> But you read: Is your data <> Should it not be just: <> ? Sent from Outlook _____________________________ From: Bogdan Andu Sent: Thursday, November 5, 2015 2:26 p.m. Subject: [erlang-questions] blowfish cbc mode decrypt To: Erlang Hi, I have a stream of data called EncryptedData (with randomiv header specification). I want to decrypt? using: crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). where: EncryptedData is formatted as follows: <> IvSpec == <<"RandomIV">> However the decryption does not works and garbage is generated (the function? ) . What could be done given the fact that IV is an 8-byte quantity the Key is formatted according to crypto test suites and is 16 bytes long Am I missing something? Thanks, Bogdan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Thu Nov 5 14:51:56 2015 From: bog495@REDACTED (Bogdan Andu) Date: Thu, 5 Nov 2015 15:51:56 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: Message-ID: Data to be decrypted (arrived on socket, etc) is: <> = EncryptedData IV is ectracted form the 8-byte prepanded value preceding IvSpec which has the value RandomIV. decryption of Rest should be initialized with IV. On Thu, Nov 5, 2015 at 3:38 PM, Radoslaw Gruchalski wrote: > You say your data is: > > <> == <> > > But you read: > > Is your data <> > > Should it not be just: > > <> > > ? > > Sent from Outlook > > _____________________________ > From: Bogdan Andu > Sent: Thursday, November 5, 2015 2:26 p.m. > Subject: [erlang-questions] blowfish cbc mode decrypt > To: Erlang > > > > Hi, > > I have a stream of data called EncryptedData (with randomiv header > specification). > I want to decrypt using: > crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). > > where: > > EncryptedData is formatted as follows: > <> > > IvSpec == <<"RandomIV">> > > However the decryption does not works and garbage is generated > (the function ) . > > What could be done given the fact that IV is an 8-byte quantity > the Key is formatted according to crypto test suites and is 16 bytes long > > Am I missing something? > > Thanks, > > Bogdan > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Nov 5 16:16:15 2015 From: g@REDACTED (Garrett Smith) Date: Thu, 5 Nov 2015 15:16:15 +0000 Subject: [erlang-questions] [ANN] Chicago Erlang meetup on patterns - Nov 12 Message-ID: If you're in Chicago next Thursday evening (Nov 12) and are looking for something Erlang to do, you're in luck! http://www.meetup.com/ErlangChicago/events/226479428/ This month we're going to spend some time on moving the Erlang Patterns project forward: http://www.erlangpatterns.org Garrett From ameretat.reith@REDACTED Thu Nov 5 17:34:29 2015 From: ameretat.reith@REDACTED (Ameretat Reith) Date: Thu, 5 Nov 2015 20:04:29 +0330 Subject: [erlang-questions] gen_sctp fail with EAGAIN, Is SCTP sockets non-blocking? Message-ID: <20151105200429.25a8b704@gmail.com> Hi, I get very poor throughput when using SCTP instead of TCP. My program simply asks another node his records in an ETS. I narrowed problem, removed pagination and tried to continuously send data to socket and saw gen_sctp:send being failed with EAGAIN after some writes. Does Erlang open sctp sockets asynchronously? Is there a way I can change it? From mremond@REDACTED Thu Nov 5 23:15:28 2015 From: mremond@REDACTED (=?utf-8?Q?Micka=C3=ABl_R=C3=A9mond?=) Date: Thu, 5 Nov 2015 23:15:28 +0100 Subject: [erlang-questions] Advanced Erlang Workshop Quickcheck videos Message-ID: Hello, We have started posting videos of Quviq Advanced Erlang Workshop in Gothenburg. The first video is a very advanced and still entertaining talk from Torben Hoffmann, Mocking for the Game of Life: http://advanced-erlang.com/news/2015-11-6-quickcheck-workshop-1-1/ More to come soon. Enjoy ! -- Micka?l R?mond From bjorn@REDACTED Fri Nov 6 11:17:21 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Fri, 6 Nov 2015 11:17:21 +0100 Subject: [erlang-questions] New EEP 45 - FUNCTION macro In-Reply-To: <1581D0E3-EE04-43FF-8F81-935A54A2740F@cs.otago.ac.nz> References: <1581D0E3-EE04-43FF-8F81-935A54A2740F@cs.otago.ac.nz> Message-ID: (I've been home with a cold all week.) On Mon, Nov 2, 2015 at 2:59 AM, Richard A. O'Keefe wrote: > For Erlang, I respectfully suggest that it makes more sense > to have *two* macros: > > ?FUNCTION_NAME => the name of the containing function > ?FUNCTION_ARITY => the arity of the containing function > That makes sense for the reasons you mentioned in your email. I will revise the EEP. /Bj?rn From bjorn@REDACTED Fri Nov 6 11:34:29 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Fri, 6 Nov 2015 11:34:29 +0100 Subject: [erlang-questions] Fourth draft of EEP 44 - Additional preprocessor directives In-Reply-To: <8018932F-D641-42F5-BB97-2F7CF7D017B4@cs.otago.ac.nz> References: <8018932F-D641-42F5-BB97-2F7CF7D017B4@cs.otago.ac.nz> Message-ID: On Mon, Nov 2, 2015 at 4:13 AM, Richard A. O'Keefe wrote: [...] > In fact this proposal *still* does not give us a way to > skip one or more clauses that *would* be syntactically > incorrect in some context. > > There is a very simple change to Erlang that would make > this possible: allow '.' between function clauses as > well as ';'. > > That would also enable us to paste a function together > out of pieces held in separate files: > > % Define f/3 > -include("old_f.hrl"). > -include("new_f.hrl"). > That change to the language should be described in a separate EEP. It can be implemented independently of any changes to the preprocessor. [...] > I do accept that 'supported' is something that can well be > added in a later EEP, and I would not take the clunkiness > of OTP_RELEASE as a show-stopper. But the documentation > should insist that anyone using OTP_RELEASE write an > explicit comment clearly stating what they are really > testing for. Yes, I think supported/1 is sufficiently complex that it will need its own EEP. I have not written any user documentation for any of the features in this EEP. Obviously the user documentation should contain disclaimers and warnings about the traps and pitfalls of the preprocessor, and examples of best practices when you have no other choice than to use the preprocessor. [...] > There is a way to test if a *header* exists (is_header/1), > which looks in the header search path. > There is a way to test if a (compiled) *module* exists > (is_module/1), which looks in the module search path. > I am convinced that these are good ideas. If only C had them! > > But there is no way to test if a *data file* exists. Do you have a concrete use case? I have tried to find real-world use cases for all built-in 'if' functions and avoided to add functions that are just "nice to have". /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From bchesneau@REDACTED Fri Nov 6 12:59:40 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 06 Nov 2015 11:59:40 +0000 Subject: [erlang-questions] [ann] just released hooks 1.1 Message-ID: Hi all, I just released hooks, a generic hooks system for #Erlang applications: https://github.com/barrel-db/hooks It allows you to augment your application by adding hooks to your application aka Hooking. Hooks' list is compiled to a beam once an hook or a list of hooks is registered using merl. The list of registered hooks can then be shared between every process of your application without message passing. It is also memory efficient and minimize locking. Hooks is also available via hex (https://hex.pm) . Hope it can be useful, Enjoy! -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Nov 6 17:00:02 2015 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 6 Nov 2015 17:00:02 +0100 Subject: [erlang-questions] Why Erlang spawns sub-processes in their own process session / group? In-Reply-To: References: Message-ID: On Tue, Oct 27, 2015 at 7:19 PM, Ciprian Dorin Craciun < ciprian.craciun@REDACTED> wrote: > > I wonder why such a decision to put newly spawned processes in new > sessions / groups than the parent Erlang VM? What is the advantage? > The original decision was made a long time ago (at least before R7B, which is as far back as I can check) and I think it may have something to do with the way terminals work. I found this blog post talking about the issue: https://blog.nelhage.com/2011/02/changing-ctty/. If you want an alternative that is more flexible you can use https://github.com/saleyn/erlexec, it appears to take care of this case by allowing you to pass an option specifying if the child has a terminal or not. Regardless, all programs that are started with open_port({spawn}) should anyway listen for a closed stdin/stdout fd in order to determine if the port that handles it has terminated, so if you take care of that case then the case of the emulator terminating takes care of itself. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From povetkin.k@REDACTED Fri Nov 6 21:24:14 2015 From: povetkin.k@REDACTED (Constantine Povietkin) Date: Fri, 6 Nov 2015 22:24:14 +0200 Subject: [erlang-questions] Test if function has been called In-Reply-To: References: <20151104003816.06c78538@kappac.home.local> Message-ID: <20151106222414.56b3821a@kappac.home.local> Thank you a ton I'll give it a try! ? Wed, 4 Nov 2015 09:47:03 +0100 Paul Peregud ?????: > You can use tracing. Redbug from eper is pretty easy to use. > Documentation is in redbug.erl file. > > https://github.com/massemanet/eper > > On Tue, Nov 3, 2015 at 11:38 PM, Constantine Povietkin > wrote: > > Hi all, > > > > I know my question has been discussed _a lot of_ times, but I've no > > luck to find an answer. > > > > I've a gen_server based module that should call handlers with apply. > > Method that calls handlers should not return anything by design. > > > > What is(are) way(s) to test if function has been called. > > > > Thanks in advance! > > > > P.S. Links to tools/documentation are highly appreciated. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > From zhuo.quan@REDACTED Sat Nov 7 12:44:15 2015 From: zhuo.quan@REDACTED (=?gb2312?B?yKvXvw==?=) Date: Sat, 7 Nov 2015 11:44:15 +0000 Subject: [erlang-questions] How to delete records in a mnesia table when traverse it Message-ID: Hi, list: I have a mnesia table about 15K records in my application. As the time goes, I find that some records(10K) in the table is useless. So I want to delete these useless items with a piece of code like this: Here is the code: %%%%%%%%%%%%%%%%%%%%% 1. delete() -> 2. traversal_tab(tab). 3. traversal_tab(TabName) -> 4. traversal_tab(TabName, mnesia:dirty_first(TabName)) . 5. traversal_tab(TabName, '$end_of_table') -> 6. io:format("finished~n"); 7. traversal_tab(TabName, Key) -> 8. delete_entry(TabName, Key), 9. traversal_tab(TabName, mnesia:dirty_next(TabName, Key)). 10. delete_entry(TabName, Key) -> 11. if ... %% delete the record if it fulfils the condition. 12. mnesia:dirty_delete(TabName, Key) ...13. end. %%%%%%%%%%%%%%%%%%%%%%%% The problem is if I delete the certain record in line 12, the function mnesia:dirty_next(TabName, Key) in line 9 will fail. How to delete records when i traverse a mnesia table? Anyone Help? Thank you ! Regards Quan Zhuo -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sat Nov 7 15:24:44 2015 From: eriksoe@REDACTED (=?UTF-8?Q?Erik_S=C3=B8e_S=C3=B8rensen?=) Date: Sat, 7 Nov 2015 15:24:44 +0100 Subject: [erlang-questions] How to delete records in a mnesia table when traverse it In-Reply-To: References: Message-ID: How about calculating dirty_next() *before* deleting the item? Den 07/11/2015 12.44 skrev "??" : > Hi, list: > > I have a mnesia table about 15K records in my application. As the time > goes, I find that some records(10K) in the table is useless. So I want to > delete these useless items with a piece of code like this: > > Here is the code: > %%%%%%%%%%%%%%%%%%%%% > 1. delete() -> > 2. traversal_tab(tab). > > 3. traversal_tab(TabName) -> > 4. traversal_tab(TabName, mnesia:dirty_first(TabName)) . > > 5. traversal_tab(TabName, '$end_of_table') -> > 6. io:format("finished~n"); > 7. traversal_tab(TabName, Key) -> > 8. delete_entry(TabName, Key), > 9. traversal_tab(TabName, mnesia:dirty_next(TabName, Key)). > > 10. delete_entry(TabName, Key) -> > 11. if ... > %% delete the record if it fulfils the condition. > 12. mnesia:dirty_delete(TabName, Key) > ... > 13. end. > %%%%%%%%%%%%%%%%%%%%%%%% > > The problem is if I delete the certain record in line 12, the function > mnesia:dirty_next(TabName, Key) in line 9 will fail. > How to delete records when i traverse a mnesia table? Anyone Help? > > Thank you ! > > Regards > Quan Zhuo > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.rosenblum@REDACTED Sun Nov 8 14:17:32 2015 From: jim.rosenblum@REDACTED (Jim Rosenblum) Date: Sun, 8 Nov 2015 08:17:32 -0500 Subject: [erlang-questions] (no subject) Message-ID: <7559FCA3-E1EB-4069-BF13-F52CA08453AC@gmail.com> From: ?? To: Erlang???? Subject: [erlang-questions] How to delete records in a mnesia table when traverse it Message-ID: Content-Type: text/plain; charset="gb2312" > Hi, list: > > I have a mnesia table about 15K records in my application. As the time goes, I find that some records(10K) in the table is useless. So I want to delete these useless items with a piece of code like this: > > Here is the code: > %%%%%%%%%%%%%%%%%%%%% > 1. delete() -> > 2. traversal_tab(tab). > > 3. traversal_tab(TabName) -> > 4. traversal_tab(TabName, mnesia:dirty_first(TabName)) . > > 5. traversal_tab(TabName, '$end_of_table') -> > 6. io:format("finished~n"); > 7. traversal_tab(TabName, Key) -> > 8. delete_entry(TabName, Key), > 9. traversal_tab(TabName, mnesia:dirty_next(TabName, Key)). > > 10. delete_entry(TabName, Key) -> > 11. if ... > %% delete the record if it fulfils the condition. > 12. mnesia:dirty_delete(TabName, Key) > ...13. end. > %%%%%%%%%%%%%%%%%%%%%%%% > > The problem is if I delete the certain record in line 12, the function mnesia:dirty_next(TabName, Key) in line 9 will fail. > How to delete records when i traverse a mnesia table? Anyone Help? > Depending on your criteria, maybe you could use dirty-match or dirty-select to collect the records that need to be deleted in one pass, and then delete those objects by iterating though that collection. Alternately, you could have a process that (at regular intervals) walks the table marking records for deletion (by setting a field reserved for this purpose) that meet your criteria and then deletes them as described above. -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Mon Nov 9 04:23:40 2015 From: serge@REDACTED (Serge Aleynikov) Date: Sun, 8 Nov 2015 22:23:40 -0500 Subject: [erlang-questions] gen_sctp fail with EAGAIN, Is SCTP sockets non-blocking? In-Reply-To: <20151105200429.25a8b704@gmail.com> References: <20151105200429.25a8b704@gmail.com> Message-ID: A few years back when we wrote the SCTP protocol binding, we did limited performance testing and it was a bit slower than of TCP but was comparable. It's is normal that you may be getting EAGAIN on the low-level sockets as all sockets are opened in the non-blocking mode. You may want to increase the sending/receiving socket buffer size, which may help a bit (see inet:setopts/2: {sndbuf, Size}, {recbuf, Size}). Serge On Thu, Nov 5, 2015 at 11:34 AM, Ameretat Reith wrote: > Hi, > > I get very poor throughput when using SCTP instead of TCP. My program > simply asks another node his records in an ETS. I narrowed problem, > removed pagination and tried to continuously send data to socket and > saw gen_sctp:send being failed with EAGAIN after some writes. > > Does Erlang open sctp sockets asynchronously? Is there a way I can > change it? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From radek@REDACTED Mon Nov 9 10:40:52 2015 From: radek@REDACTED (Rad Gruchalski) Date: Mon, 9 Nov 2015 10:40:52 +0100 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: Message-ID: <69926156260A47EF992A42555D468AFA@gruchalski.com> Bogdan, I wonder if you can show us a minimum viable example. How do you encrypt, how do you decrypt. Can you present such code? Kind regards, Radek Gruchalski radek@REDACTED (mailto:radek@REDACTED) (mailto:radek@REDACTED) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Monday, 9 November 2015 at 10:38, Bogdan Andu wrote: > > hi, > > I really need some pointers about this issue > > as i tried all kinds of combinations of IV and body > to be decrypted. > > Am i missing something or there is a bug in cipher > blowfish_cbc ? > > > > On Thu, Nov 5, 2015 at 3:51 PM, Bogdan Andu wrote: > > Data to be decrypted (arrived on socket, etc) is: > > <> = EncryptedData > > > > IV is ectracted form the 8-byte prepanded value preceding > > IvSpec which has the value RandomIV. > > > > decryption of Rest should be initialized with IV. > > > > > > On Thu, Nov 5, 2015 at 3:38 PM, Radoslaw Gruchalski wrote: > > > You say your data is: > > > > > > <> == <> > > > > > > But you read: > > > > > > Is your data <> > > > > > > Should it not be just: > > > > > > <> > > > > > > ? > > > > > > Sent from Outlook (http://aka.ms/Ox5hz3) > > > _____________________________ > > > From: Bogdan Andu > > > Sent: Thursday, November 5, 2015 2:26 p.m. > > > Subject: [erlang-questions] blowfish cbc mode decrypt > > > To: Erlang > > > > > > > > > > > > Hi, > > > > > > I have a stream of data called EncryptedData (with randomiv header specification). > > > I want to decrypt using: > > > crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). > > > > > > where: > > > > > > EncryptedData is formatted as follows: > > > <> > > > > > > IvSpec == <<"RandomIV">> > > > > > > However the decryption does not works and garbage is generated > > > (the function ) . > > > > > > What could be done given the fact that IV is an 8-byte quantity > > > the Key is formatted according to crypto test suites and is 16 bytes long > > > > > > Am I missing something? > > > > > > Thanks, > > > > > > Bogdan > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Mon Nov 9 10:38:06 2015 From: bog495@REDACTED (Bogdan Andu) Date: Mon, 9 Nov 2015 11:38:06 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: Message-ID: hi, I really need some pointers about this issue as i tried all kinds of combinations of IV and body to be decrypted. Am i missing something or there is a bug in cipher blowfish_cbc ? On Thu, Nov 5, 2015 at 3:51 PM, Bogdan Andu wrote: > Data to be decrypted (arrived on socket, etc) is: > <> = EncryptedData > > IV is ectracted form the 8-byte prepanded value preceding > IvSpec which has the value RandomIV. > > decryption of Rest should be initialized with IV. > > > On Thu, Nov 5, 2015 at 3:38 PM, Radoslaw Gruchalski > wrote: > >> You say your data is: >> >> <> == <> >> >> But you read: >> >> Is your data <> >> >> Should it not be just: >> >> <> >> >> ? >> >> Sent from Outlook >> >> _____________________________ >> From: Bogdan Andu >> Sent: Thursday, November 5, 2015 2:26 p.m. >> Subject: [erlang-questions] blowfish cbc mode decrypt >> To: Erlang >> >> >> >> Hi, >> >> I have a stream of data called EncryptedData (with randomiv header >> specification). >> I want to decrypt using: >> crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). >> >> where: >> >> EncryptedData is formatted as follows: >> <> >> >> IvSpec == <<"RandomIV">> >> >> However the decryption does not works and garbage is generated >> (the function ) . >> >> What could be done given the fact that IV is an 8-byte quantity >> the Key is formatted according to crypto test suites and is 16 bytes long >> >> Am I missing something? >> >> Thanks, >> >> Bogdan >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Mon Nov 9 10:51:08 2015 From: bog495@REDACTED (Bogdan Andu) Date: Mon, 9 Nov 2015 11:51:08 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: <69926156260A47EF992A42555D468AFA@gruchalski.com> References: <69926156260A47EF992A42555D468AFA@gruchalski.com> Message-ID: I have DataToBeDecrypted and one-liner in erlang shell: crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). where DataToBeDecrypted is formatted as follows: <> = DataToBeDecrypted I can extract IvSpec = <<"RandomIV">>, Next 8 octets is IV, Rest should bet the actual quantity to be decrypted, and the function call above fails to produce the plain text expected, only garbage Unfortunately Key is sensitive and cannot disclose the value. On Mon, Nov 9, 2015 at 11:40 AM, Rad Gruchalski wrote: > Bogdan, > > I wonder if you can show us a minimum viable example. > How do you encrypt, how do you decrypt. Can you present such code? > > Kind regards, > Radek Gruchalski > radek@REDACTED > de.linkedin.com/in/radgruchalski/ > > > *Confidentiality:*This communication is intended for the above-named > person and may be confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor > must you copy or show it to anyone; please delete/destroy and inform the > sender immediately. > > On Monday, 9 November 2015 at 10:38, Bogdan Andu wrote: > > > hi, > > I really need some pointers about this issue > > as i tried all kinds of combinations of IV and body > to be decrypted. > > Am i missing something or there is a bug in cipher > blowfish_cbc ? > > > > On Thu, Nov 5, 2015 at 3:51 PM, Bogdan Andu wrote: > > Data to be decrypted (arrived on socket, etc) is: > <> = EncryptedData > > IV is ectracted form the 8-byte prepanded value preceding > IvSpec which has the value RandomIV. > > decryption of Rest should be initialized with IV. > > > On Thu, Nov 5, 2015 at 3:38 PM, Radoslaw Gruchalski > wrote: > > You say your data is: > > <> == <> > > But you read: > > Is your data <> > > Should it not be just: > > <> > > ? > > Sent from Outlook > > _____________________________ > From: Bogdan Andu > Sent: Thursday, November 5, 2015 2:26 p.m. > Subject: [erlang-questions] blowfish cbc mode decrypt > To: Erlang > > > > Hi, > > I have a stream of data called EncryptedData (with randomiv header > specification). > I want to decrypt using: > crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). > > where: > > EncryptedData is formatted as follows: > <> > > IvSpec == <<"RandomIV">> > > However the decryption does not works and garbage is generated > (the function ) . > > What could be done given the fact that IV is an 8-byte quantity > the Key is formatted according to crypto test suites and is 16 bytes long > > Am I missing something? > > Thanks, > > Bogdan > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bog495@REDACTED Mon Nov 9 14:17:56 2015 From: bog495@REDACTED (Bogdan Andu) Date: Mon, 9 Nov 2015 15:17:56 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: <69926156260A47EF992A42555D468AFA@gruchalski.com> References: <69926156260A47EF992A42555D468AFA@gruchalski.com> Message-ID: I have DataToBeDecrypted and one-liner in erlang shell: crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). where DataToBeDecrypted is formatted as follows: <> = DataToBeDecrypted I can extract IvSpec = <<"RandomIV">>, Next 8 octets is IV, Rest should bet the actual quantity to be decrypted, and the function call above fails to produce the plain text expected, only garbage Unfortunately Key is sensitive and cannot disclose the value. On Mon, Nov 9, 2015 at 11:40 AM, Rad Gruchalski wrote: > Bogdan, > > I wonder if you can show us a minimum viable example. > How do you encrypt, how do you decrypt. Can you present such code? > > Kind regards, > Radek Gruchalski > radek@REDACTED > de.linkedin.com/in/radgruchalski/ > > > *Confidentiality:*This communication is intended for the above-named > person and may be confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor > must you copy or show it to anyone; please delete/destroy and inform the > sender immediately. > > On Monday, 9 November 2015 at 10:38, Bogdan Andu wrote: > > > hi, > > I really need some pointers about this issue > > as i tried all kinds of combinations of IV and body > to be decrypted. > > Am i missing something or there is a bug in cipher > blowfish_cbc ? > > > > On Thu, Nov 5, 2015 at 3:51 PM, Bogdan Andu wrote: > > Data to be decrypted (arrived on socket, etc) is: > <> = EncryptedData > > IV is ectracted form the 8-byte prepanded value preceding > IvSpec which has the value RandomIV. > > decryption of Rest should be initialized with IV. > > > On Thu, Nov 5, 2015 at 3:38 PM, Radoslaw Gruchalski > wrote: > > You say your data is: > > <> == <> > > But you read: > > Is your data <> > > Should it not be just: > > <> > > ? > > Sent from Outlook > > _____________________________ > From: Bogdan Andu > Sent: Thursday, November 5, 2015 2:26 p.m. > Subject: [erlang-questions] blowfish cbc mode decrypt > To: Erlang > > > > Hi, > > I have a stream of data called EncryptedData (with randomiv header > specification). > I want to decrypt using: > crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). > > where: > > EncryptedData is formatted as follows: > <> > > IvSpec == <<"RandomIV">> > > However the decryption does not works and garbage is generated > (the function ) . > > What could be done given the fact that IV is an 8-byte quantity > the Key is formatted according to crypto test suites and is 16 bytes long > > Am I missing something? > > Thanks, > > Bogdan > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Mon Nov 9 14:55:00 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 9 Nov 2015 14:55:00 +0100 Subject: [erlang-questions] EEP 45: New macros for function name and arity Message-ID: Second draft of EEP 45. The FUNCTION macro has been replaced with FUNCTION_NAME and FUNCTION_ARITY macro. http://www.erlang.org/eeps/eep-0045.html https://github.com/erlang/eep/blob/master/eeps/eep-0045.md -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From ivan@REDACTED Mon Nov 9 16:10:45 2015 From: ivan@REDACTED (Ivan Uemlianin) Date: Mon, 9 Nov 2015 15:10:45 +0000 Subject: [erlang-questions] Languages running on BEAM Message-ID: <5640B775.1090700@llaisdy.com> Dear All I am compiling a list of programming languages that run on BEAM, the Erlang VM: https://github.com/llaisdy/beam_languages The resource is aimed at people who might be thinking of implementing such a language themselves. Consequently the list includes experimental and abandoned languages as well as the stars. I am also including a section listing documents (papers, presentations, blog posts, etc.) that cover the theme of implementing languages for the BEAM in general. Languages listed so far: - efene - elixir - erl2 - erlang - erlog - erlyjs - haskerl - joxa - lfe - luerl - reia - 'the concurrent schemer' - xerl If you know about a language or a document that should be on one of these lists --- or if I've made mistakes! ---, please let me know (email, issue/PR, tweet, etc.). With thanks and best wishes Ivan -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED @llaisdy llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From sverker.eriksson@REDACTED Mon Nov 9 16:21:48 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 9 Nov 2015 16:21:48 +0100 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: <69926156260A47EF992A42555D468AFA@gruchalski.com> Message-ID: <5640BA0C.8080109@ericsson.com> The call crypto:block_decrypt(blowfish_cbc, ... ends up in C-function bf_cbc_crypt() in $ERL_TOP/lib/crypto/c_src/crypto.c which in turn calls OpenSSL functions BF_set_key() and BF_cbc_encrypt(). If you know the exact plain text to expect, have you tried doing the reverse with crypto:block_encrypt. What about trying the same with an unsensitive key that you can share. Still don't understand what role IvSpec plays in all this. /Sverker, Erlang/OTP On 11/09/2015 02:17 PM, Bogdan Andu wrote: > I have DataToBeDecrypted > > and one-liner in erlang shell: > > crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). > > where DataToBeDecrypted is formatted as follows: > > <> = DataToBeDecrypted > > I can extract IvSpec = <<"RandomIV">>, > > Next 8 octets is IV, > > Rest should bet the actual quantity to be decrypted, > > and the function call above fails to produce the plain text expected, > only garbage > > Unfortunately Key is sensitive and cannot disclose the value. > > On Mon, Nov 9, 2015 at 11:40 AM, Rad Gruchalski > wrote: > > Bogdan, > > I wonder if you can show us a minimum viable example. > How do you encrypt, how do you decrypt. Can you present such code? > > Kind regards, > Radek Gruchalski > radek@REDACTED > > de.linkedin.com/in/radgruchalski/ > > > *Confidentiality: > *This communication is intended for the above-named person and may > be confidential and/or legally privileged. > If it has come to you in error you must take no action based on > it, nor must you copy or show it to anyone; please delete/destroy > and inform the sender immediately. > > On Monday, 9 November 2015 at 10:38, Bogdan Andu wrote: > >> >> hi, >> >> I really need some pointers about this issue >> >> as i tried all kinds of combinations of IV and body >> to be decrypted. >> >> Am i missing something or there is a bug in cipher >> blowfish_cbc ? >> >> >> >> On Thu, Nov 5, 2015 at 3:51 PM, Bogdan Andu > > wrote: >>> Data to be decrypted (arrived on socket, etc) is: >>> <> = EncryptedData >>> >>> IV is ectracted form the 8-byte prepanded value preceding >>> IvSpec which has the value RandomIV. >>> >>> decryption of Rest should be initialized with IV. >>> >>> >>> On Thu, Nov 5, 2015 at 3:38 PM, Radoslaw Gruchalski >>> > wrote: >>>> You say your data is: >>>> >>>> <> == <> >>>> >>>> But you read: >>>> >>>> Is your data <> >>>> >>>> Should it not be just: >>>> >>>> <> >>>> >>>> ? >>>> >>>> Sent from Outlook >>>> >>>> _____________________________ >>>> From: Bogdan Andu > >>>> Sent: Thursday, November 5, 2015 2:26 p.m. >>>> Subject: [erlang-questions] blowfish cbc mode decrypt >>>> To: Erlang >>> > >>>> >>>> >>>> >>>> Hi, >>>> >>>> I have a stream of data called EncryptedData (with randomiv >>>> header specification). >>>> I want to decrypt using: >>>> crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). >>>> >>>> where: >>>> >>>> EncryptedData is formatted as follows: >>>> <> >>>> >>>> IvSpec == <<"RandomIV">> >>>> >>>> However the decryption does not works and garbage is generated >>>> (the function ) . >>>> >>>> What could be done given the fact that IV is an 8-byte quantity >>>> the Key is formatted according to crypto test suites and is 16 >>>> bytes long >>>> >>>> Am I missing something? >>>> >>>> Thanks, >>>> >>>> Bogdan >>>> >>>> >>> >> > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Mon Nov 9 16:32:25 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Mon, 9 Nov 2015 16:32:25 +0100 Subject: [erlang-questions] Local eunit tests affect coverage value Message-ID: Hi all, Maybe I'm missing something, but I recently started to systematically use cover and now I also moved all eunit tests for non-exported functions inside the respective module (guarded by ifdef(TEST)). The resulting coverage reports now include the test functions' code, which makes them a bit skewed. Is there any way to ignore the parts related to the tests themselves? I couldn't find any... best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean@REDACTED Mon Nov 9 18:00:02 2015 From: sean@REDACTED (Functional Jobs) Date: Mon, 9 Nov 2015 12:00:02 -0500 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <5640d13e4194d@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Software Engineer, Erlang at Klarna https://functionaljobs.com/jobs/8864-software-engineer-erlang-at-klarna Cheers, Sean Murphy FunctionalJobs.com From dmkolesnikov@REDACTED Mon Nov 9 18:06:17 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Mon, 9 Nov 2015 19:06:17 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: Message-ID: Hello, I would double check that you are * using same key on both side * IV is properly serialized and de-serialize to packet frame BTW, What is the size of Rest? and Can you share minimal viable example of encryption? - Dmitry > On Nov 9, 2015, at 11:38 AM, Bogdan Andu wrote: > > > hi, > > I really need some pointers about this issue > > as i tried all kinds of combinations of IV and body > to be decrypted. > > Am i missing something or there is a bug in cipher > blowfish_cbc ? > > > > On Thu, Nov 5, 2015 at 3:51 PM, Bogdan Andu wrote: > Data to be decrypted (arrived on socket, etc) is: > <> = EncryptedData > > IV is ectracted form the 8-byte prepanded value preceding > IvSpec which has the value RandomIV. > > decryption of Rest should be initialized with IV. > > > On Thu, Nov 5, 2015 at 3:38 PM, Radoslaw Gruchalski wrote: > You say your data is: > > <> == <> > > But you read: > > Is your data <> > > Should it not be just: > > <> > > ? > > Sent from Outlook > > _____________________________ > From: Bogdan Andu > Sent: Thursday, November 5, 2015 2:26 p.m. > Subject: [erlang-questions] blowfish cbc mode decrypt > To: Erlang > > > > Hi, > > I have a stream of data called EncryptedData (with randomiv header specification). > I want to decrypt using: > crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). > > where: > > EncryptedData is formatted as follows: > <> > > IvSpec == <<"RandomIV">> > > However the decryption does not works and garbage is generated > (the function ) . > > What could be done given the fact that IV is an 8-byte quantity > the Key is formatted according to crypto test suites and is 16 bytes long > > Am I missing something? > > Thanks, > > Bogdan > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bog495@REDACTED Tue Nov 10 10:58:22 2015 From: bog495@REDACTED (Bogdan Andu) Date: Tue, 10 Nov 2015 11:58:22 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: Message-ID: came back with example: Encryption part is a Perl script: use strict; use warnings; use MIME::Base64; use Crypt::CBC; #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); use PHP::Serialization qw(serialize unserialize); my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl 7bap 71district XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:" https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; print $pt, "\n"; my $key = "12345678900987654321001234567890"; my $cipher = Crypt::CBC->new( -key => $key, -cipher => 'Blowfish', -header => 'randomiv' ); # print "$pt\n"; my $encpt = $cipher->encrypt($pt); print "\n$encpt", "\n"; print "\n", encode_base64($encpt), "\n"; ## TEST my $decpt = $cipher->decrypt($encpt); print "\n$decpt", "\n"; I want this (Base64 encoded) quantity decrypted in Erlang part: $ erl Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false] Eshell V7.0 (abort with ^G) 1> f(B), B="UmFuZG9tSVYGr2WfCVKA0rvdisU2mA+OGo4yopGlUsszoG+V3vXRifqRM+YkAYWzdodx0LdyYJC7cwvY+iRGytQxUBOpOjEBvvhXRFyTu88+FeWPh2nZ28BDLqYPgg5tQujWYlipdrafH6XUzgpXK8QMZ/JpgIYYHZ3b8te2iWU3YsCo76/2W6n1D8zyWSdJ9owr/KtmrYE4FqLszvg4Vj0l62Q95sXNQOBb7g0GwuwJm6CRbLY6FmVDkFCPzqKf/QZbWASfIFokmfhnHR+Rpc8y2dz+eJjJn0LeFwajcSCf78IoZ1j8BW02mcyQFji/e/TfuqImRrhFptBlvMggu6g6dksM4iCW8E63y2KvO2rCbnbIyapENcZaNtfsUkw/WQNGP72VDvZbHeb/E1NIwnbsOO5qHnzMAi2SFVXSMr+k6mIS+R5wKDvkquB7I1Vho5SVdnjE1LtaetLDPF8kFNh09ig51OPCQyezpEsIPBI+U9mWpAyZTJU7YtnZUFY4oA4tbMPUthkOE2E0qIQT7OL0hwTyHsn/YFCyKu7JqnEU9UaCItb8SteLpb4ZlGJsoPxdZPnaBJJniPhod+0a7mDuo/N+hijk6rHEs2hXeUNgSSyoMVylMAuQyqDJMuOWMWMNWlr/HjpIWiRw6h2F5rrR8peV20sNXo9MHdcWG/XbZdUlGwfkHgi23PtcO/fpkdQ3UTRmOcm3I3KG2w8YAc10434pbs2NYihBl7v4Rx5H0NkJePPpRzHsavYO8QH5R9INzUh0yYXgz5jWpQbqSp7FMVYnwLqEgLoXt0r7WQqU+9tgyyPQxfRZQd5URs5QxwoGC+zgDqgLCdNzdzanePh5IfCsDoeUJbZ3C4SAGxiZ+OPZYu8ixM/sYqjQmPgmXs1a7X1VthbNLxrSp5pm2cjyTRal+AFr". 2> 2>f(Bdec),Bdec = base64:decode(B). 3>Key = hexstr2bin("12345678900987654321001234567890"). 4><> = Bdec. 5>f(CifDec),CifDec = crypto:block_decrypt(blowfish_cbc, Key, IV, Rest). 6>io:format("~s~n", [CifDec]). ---output garbage---[should be $pt variable from Perl script] 7>erlang:size(Rest). 680 8> 680/8. 85.0 9>IvSpec. <<"RandomIV">> 10> ... Notes: hexstr2bin/1 is taken from otp_src_18.0/lib/crypto/test/crypto_SUITE.erl On Mon, Nov 9, 2015 at 7:06 PM, Dmitry Kolesnikov wrote: > Hello, > > I would double check that you are > * using same key on both side > * IV is properly serialized and de-serialize to packet frame > > BTW, What is the size of Rest? and Can you share minimal viable example of > encryption? > > - Dmitry > > > On Nov 9, 2015, at 11:38 AM, Bogdan Andu wrote: > > > > > > hi, > > > > I really need some pointers about this issue > > > > as i tried all kinds of combinations of IV and body > > to be decrypted. > > > > Am i missing something or there is a bug in cipher > > blowfish_cbc ? > > > > > > > > On Thu, Nov 5, 2015 at 3:51 PM, Bogdan Andu wrote: > > Data to be decrypted (arrived on socket, etc) is: > > <> = EncryptedData > > > > IV is ectracted form the 8-byte prepanded value preceding > > IvSpec which has the value RandomIV. > > > > decryption of Rest should be initialized with IV. > > > > > > On Thu, Nov 5, 2015 at 3:38 PM, Radoslaw Gruchalski < > radek@REDACTED> wrote: > > You say your data is: > > > > <> == <> > > > > But you read: > > > > Is your data <> > > > > Should it not be just: > > > > <> > > > > ? > > > > Sent from Outlook > > > > _____________________________ > > From: Bogdan Andu > > Sent: Thursday, November 5, 2015 2:26 p.m. > > Subject: [erlang-questions] blowfish cbc mode decrypt > > To: Erlang > > > > > > > > Hi, > > > > I have a stream of data called EncryptedData (with randomiv header > specification). > > I want to decrypt using: > > crypto:block_decrypt(blowfish_cbc, hexstr2bin(Key), IV, Rest). > > > > where: > > > > EncryptedData is formatted as follows: > > <> > > > > IvSpec == <<"RandomIV">> > > > > However the decryption does not works and garbage is generated > > (the function ) . > > > > What could be done given the fact that IV is an 8-byte quantity > > the Key is formatted according to crypto test suites and is 16 bytes long > > > > Am I missing something? > > > > Thanks, > > > > Bogdan > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Tue Nov 10 12:15:40 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 10 Nov 2015 13:15:40 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: Message-ID: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com> Hello, Yes, this want I thought. You are using wrong key to decrypt data on Erlang side. Please notice that ?Crypt::CBC can derive the key and the IV from a passphrase that you provide, or can let you specify the true key and IV manually? The -key argument provides either a passphrase to use to generate the encryption key, or the literal value of the block cipher key. If used in passphrase mode (which is the default), -key can be any number of characters; the actual key will be derived by passing the passphrase through a series of MD5 hash operations.? So, in your example Key is not an encryption key, this is a pass-phrase. The Erlang?s implementation expects that you provides actual key. I do not know how perl?s Crypt::CBC derives the key from perspires. You can either reverse engineer that piece of code or you can use literal key. Best Regards, Dmitry > On Nov 10, 2015, at 11:58 AM, Bogdan Andu wrote: > > use strict; > use warnings; > use MIME::Base64; > > use Crypt::CBC; > #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); > use PHP::Serialization qw(serialize unserialize); > > my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl 7bap 71district XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:"https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; > > print $pt, "\n"; > > my $key = "12345678900987654321001234567890"; > my $cipher = Crypt::CBC->new( > -key => $key, > -cipher => 'Blowfish', > -header => 'randomiv' > ); > > # print "$pt\n"; > my $encpt = $cipher->encrypt($pt); > print "\n$encpt", "\n"; > > print "\n", encode_base64($encpt), "\n"; > > ## TEST > my $decpt = $cipher->decrypt($encpt); > print "\n$decpt", "\n"; From bog495@REDACTED Tue Nov 10 14:51:24 2015 From: bog495@REDACTED (Bogdan Andu) Date: Tue, 10 Nov 2015 15:51:24 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com> References: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com> Message-ID: I was thinking that is handled internally... however, this did the trick (for randomiv mode of operation): blowfish_key_from_key(Key) -> Temp = crypto:md5(Key), blowfish_key_from_key1(Temp). blowfish_key_from_key1(Temp) when size(Temp) < 56 -> Temp1 = crypto:md5(Temp), blowfish_key_from_key1(<>); blowfish_key_from_key1(Temp) -> <> = Temp, Temp1. I think that crypto should have this kind oh helper functions because , like in cases of blowfish cipher , whose key length can be variable, could be useful in using the cipher correctly, and people would know that some things are not implied. Thanks, Bogdan On Tue, Nov 10, 2015 at 1:15 PM, Dmitry Kolesnikov wrote: > Hello, > > Yes, this want I thought. You are using wrong key to decrypt data on > Erlang side. > > Please notice that > > ?Crypt::CBC can derive the key and the IV from a passphrase that you > provide, or can let you specify the true key and IV manually? > > The -key argument provides either a passphrase to use to generate the > encryption key, or the literal value of the block cipher key. If used in > passphrase mode (which is the default), -key can be any number of > characters; the actual key will be derived by passing the passphrase > through a series of MD5 hash operations.? > > So, in your example Key is not an encryption key, this is a pass-phrase. > The Erlang?s implementation expects that you provides actual key. > > I do not know how perl?s Crypt::CBC derives the key from perspires. You > can either reverse engineer that piece of code or you can use literal key. > > > Best Regards, > Dmitry > > > > On Nov 10, 2015, at 11:58 AM, Bogdan Andu wrote: > > > > use strict; > > use warnings; > > use MIME::Base64; > > > > use Crypt::CBC; > > #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); > > use PHP::Serialization qw(serialize unserialize); > > > > my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST > \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl > 7bap 71district > XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:" > https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr > 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 > \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 > \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 > \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr > 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; > > > > print $pt, "\n"; > > > > my $key = "12345678900987654321001234567890"; > > my $cipher = Crypt::CBC->new( > > -key => $key, > > -cipher => 'Blowfish', > > -header => 'randomiv' > > ); > > > > # print "$pt\n"; > > my $encpt = $cipher->encrypt($pt); > > print "\n$encpt", "\n"; > > > > print "\n", encode_base64($encpt), "\n"; > > > > ## TEST > > my $decpt = $cipher->decrypt($encpt); > > print "\n$decpt", "\n"; > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ameretat.reith@REDACTED Tue Nov 10 15:57:42 2015 From: ameretat.reith@REDACTED (Ameretat Reith) Date: Tue, 10 Nov 2015 18:27:42 +0330 Subject: [erlang-questions] Remote shell and terminfo Message-ID: I faced strange things with different values in TERM environment variable and remote_shell. First, If I try open a remote shell with a defined terminfo -understood by installed version of ncurses- I get a crash: ``` $ TERM=xterm erl -remsh nodename Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false] *** ERROR: Shell process terminated! (^G to start new job) *** ``` Which I thought may be because of different versions of running erts (7.0) and that tries to make remote shell (5.10.4), but I can make a remote shell If I set TERM to something garbage: ``` $ TERM=screen-blah-blah erl -remsh nodename Eshell V5.10.4 (abort with ^G) 1> proplists:get_value(max_fds, erlang:system_info(check_io)). 1024 ``` Using same Erlang version as provided in made release by `relx` and using start scripts, I can make remote console with or without known terminfo: ``` $ TERM=screen-blah-blah /path/to/rel/bin/start remote_console Eshell V7.0 (abort with ^G) (remsh2a0a97fb-nodename@REDACTED:247)1> proplists:get_value(max_fds, erlang:system_info(check_io)). 1024 ``` And things get funny... ``` $ TERM=xterm /path/to/rel/bin/start remote_console Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] Eshell V7.0 (abort with ^G) (nodename@REDACTED:247)1> proplists:get_value(max_fds, erlang:system_info(check_io)). 262144 ``` I get different maximum file descriptor limit with different terminfo. Looks like last one is not really a connected remote shell.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From francesco@REDACTED Tue Nov 10 18:51:55 2015 From: francesco@REDACTED (Francesco Cesarini) Date: Tue, 10 Nov 2015 17:51:55 +0000 Subject: [erlang-questions] 2016 OSCON CFP Deadline 24/11 (Ausitn, Tx) Message-ID: <56422EBB.4080705@erlang-solutions.com> Hi All, next year, OSCON, the open source convention, is moving to Texas. Austin, to be exact. Dates are May 18th and 19th. We are hoping in having a large Erlang presence as usual, so it would be great if you were consider submitting. Deadline is November 24th. There are dozens of tracks and target groups, so lots of opportunities for everyone. Tutorials are also welcome. More info is here: http://conferences.oreilly.com/oscon/open-source/public/cfp/423 Help making Erlang, Elixir and LFE visible! Francesco From lloyd@REDACTED Tue Nov 10 19:24:43 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 10 Nov 2015 13:24:43 -0500 (EST) Subject: [erlang-questions] Illusive mnesia bug Message-ID: <1447179883.352530994@apps.rackspace.com> Hello, I've been building a simple mnesia-based app--- basically a prototype to gain experience with mnesia. See nim_links1.erl, attached. Had it working, but following a few simple but undocumented/unremembered changes, it no longer works. So far I've spent a day trying to understand how I managed to break it. Here's what I know: 1. nim_links1.erl compiles 2. mnesia is running and links table is installed: 40> mnesia:info(). ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- schema : with 2 records occupying 533 words of mem links : with 0 records occupying 299 words of mem ===> System info in version "4.13.1", debug level = none <=== opt_disc. Directory "/home/lloyd/nindex/Mnesia.nindex@REDACTED" is used. use fallback at restart = false running db nodes = ['nindex@REDACTED'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [links,schema] disc_only_copies = [] [{'nindex@REDACTED',disc_copies}] = [links,schema] 4 transactions committed, 6 aborted, 0 restarted, 0 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] ok 3. record_info/2 works: 41> record_info(fields, weblink). [id,topic,tags,descriptor,url] 4. As do nim_links1:create_link_record/4 and nim_links1:get_all/0: 42> Topic = "Erlang". "Erlang" 43> Tags = []. [] 44> Descriptor = "Nerves Project (embedded Erlang)". "Nerves Project (embedded Erlang)" 45> Url = "http://nerves-project.org/". "http://nerves-project.org/" 46> Record = nim_links1:create_link_record(Topic, Tags, Descriptor, Url). #weblink{id = "ECFB6AB4443",topic = "Erlang",tags = [], descriptor = "Nerves Project (embedded Erlang)", url = "http://nerves-project.org/"} 48> nim_links1:get_all(). [] 5. BUT, nim_links1/1 fails: 50> nim_links1:put_record(Record). ** exception error: no match of right hand side value {aborted,{no_exists,weblink}} in function nim_links1:put_record/1 (/home/lloyd/nindex/site/src /nim_links1.erl, line 50) 6. As does mnesia:dirty_write/1: 51> mnesia:dirty_write(Record). ** exception exit: {aborted,{no_exists,weblink}} in function mnesia:abort/1 (mnesia.erl, line 313) And now I've run out of ideas of how to proceed. If I'm reading the error messages correctly, the mnesia write functions are not recognizing the defined weblink record. I've attached ni_mnesia.erl to show how I initialized mnesia. I'd much appreciate any insights into - where I'm going wrong and how to fix it - debugging strategies in cases like this - how I can improve the code Many thanks, LRP -------------- next part -------------- A non-text attachment was scrubbed... Name: nim_links1.erl Type: text/x-erlang Size: 1445 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ni_mnesia.erl Type: text/x-erlang Size: 1108 bytes Desc: not available URL: From jesper.louis.andersen@REDACTED Tue Nov 10 19:42:32 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 10 Nov 2015 19:42:32 +0100 Subject: [erlang-questions] Illusive mnesia bug In-Reply-To: <1447179883.352530994@apps.rackspace.com> References: <1447179883.352530994@apps.rackspace.com> Message-ID: On Tue, Nov 10, 2015 at 7:24 PM, wrote: > 5. BUT, nim_links1/1 fails: > > 50> nim_links1:put_record(Record). > > ** exception error: no match of right hand side value > {aborted,{no_exists,weblink}} > in function nim_links1:put_record/1 (/home/lloyd/nindex/site/src > /nim_links1.erl, line 50) > Your problem is that you have a record #weblink{} but the table you create is named ?TABLE which is the value 'links'. You have not supplied {record_name, weblink} so the default record name becomes the same as the table name. The error mnesia reports is because there is no table in which to place #weblink{} records, but you should have one named links. Test with `mnesia:table_info(links, all)`. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Nov 10 20:44:01 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 10 Nov 2015 22:44:01 +0300 Subject: [erlang-questions] enif_send and smp/non-smp beam: how to check dynamically? Message-ID: enif_send: env==NULL on non-SMP VM How to check dynamically if erlang is smp or non smp and give or not give env? -------------- next part -------------- An HTML attachment was scrubbed... URL: From constantin@REDACTED Tue Nov 10 20:49:44 2015 From: constantin@REDACTED (Constantin Rack) Date: Tue, 10 Nov 2015 20:49:44 +0100 Subject: [erlang-questions] enif_send and smp/non-smp beam: how to check dynamically? In-Reply-To: References: Message-ID: <172BA26F-0028-4E42-8B7C-17CAB454B299@rack.li> To check for SMP, use: > erlang:system_info(smp_support). http://www.erlang.org/doc/man/erlang.html#system_info-1 -Constantin Am 10.11.2015 um 20:44 schrieb Max Lapshin: > enif_send: env==NULL on non-SMP VM > > > > How to check dynamically if erlang is smp or non smp and give or not give env? > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Tue Nov 10 20:58:43 2015 From: vinoski@REDACTED (Steve Vinoski) Date: Tue, 10 Nov 2015 14:58:43 -0500 Subject: [erlang-questions] enif_send and smp/non-smp beam: how to check dynamically? In-Reply-To: <172BA26F-0028-4E42-8B7C-17CAB454B299@rack.li> References: <172BA26F-0028-4E42-8B7C-17CAB454B299@rack.li> Message-ID: On Tue, Nov 10, 2015 at 2:49 PM, Constantin Rack wrote: > To check for SMP, use: > > > erlang:system_info(smp_support). > > http://www.erlang.org/doc/man/erlang.html#system_info-1 > Or in a NIF, call enif_system_info() to check for smp support: http://www.erlang.org/doc/man/erl_nif.html#enif_system_info http://www.erlang.org/doc/man/erl_driver.html#ErlDrvSysInfo --steve > > -Constantin > > > Am 10.11.2015 um 20:44 schrieb Max Lapshin: > > enif_send: env==NULL on non-SMP VM > > > How to check dynamically if erlang is smp or non smp and give or not give > env? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Nov 10 21:39:24 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 10 Nov 2015 23:39:24 +0300 Subject: [erlang-questions] enif_send and smp/non-smp beam: how to check dynamically? In-Reply-To: References: <172BA26F-0028-4E42-8B7C-17CAB454B299@rack.li> Message-ID: Thanks On Tue, Nov 10, 2015 at 10:58 PM, Steve Vinoski wrote: > > > On Tue, Nov 10, 2015 at 2:49 PM, Constantin Rack > wrote: > >> To check for SMP, use: >> >> > erlang:system_info(smp_support). >> >> http://www.erlang.org/doc/man/erlang.html#system_info-1 >> > > Or in a NIF, call enif_system_info() to check for smp support: > > http://www.erlang.org/doc/man/erl_nif.html#enif_system_info > http://www.erlang.org/doc/man/erl_driver.html#ErlDrvSysInfo > > --steve > > >> >> -Constantin >> >> >> Am 10.11.2015 um 20:44 schrieb Max Lapshin: >> >> enif_send: env==NULL on non-SMP VM >> >> >> How to check dynamically if erlang is smp or non smp and give or not give >> env? >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From technion@REDACTED Wed Nov 11 00:22:17 2015 From: technion@REDACTED (Technion) Date: Tue, 10 Nov 2015 23:22:17 +0000 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com>, Message-ID: Hi, I'm not sure if you have other mitigating code, but I'd be concerned about calling two calls to MD5 on a passphrase a "derived key" - this would be very weak crypto. ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Bogdan Andu Sent: Wednesday, 11 November 2015 12:51 AM To: Dmitry Kolesnikov Cc: Erlang Subject: Re: [erlang-questions] blowfish cbc mode decrypt I was thinking that is handled internally... however, this did the trick (for randomiv mode of operation): blowfish_key_from_key(Key) -> Temp = crypto:md5(Key), blowfish_key_from_key1(Temp). blowfish_key_from_key1(Temp) when size(Temp) < 56 -> Temp1 = crypto:md5(Temp), blowfish_key_from_key1(<>); blowfish_key_from_key1(Temp) -> <> = Temp, Temp1. I think that crypto should have this kind oh helper functions because , like in cases of blowfish cipher , whose key length can be variable, could be useful in using the cipher correctly, and people would know that some things are not implied. Thanks, Bogdan On Tue, Nov 10, 2015 at 1:15 PM, Dmitry Kolesnikov > wrote: Hello, Yes, this want I thought. You are using wrong key to decrypt data on Erlang side. Please notice that "Crypt::CBC can derive the key and the IV from a passphrase that you provide, or can let you specify the true key and IV manually... The -key argument provides either a passphrase to use to generate the encryption key, or the literal value of the block cipher key. If used in passphrase mode (which is the default), -key can be any number of characters; the actual key will be derived by passing the passphrase through a series of MD5 hash operations." So, in your example Key is not an encryption key, this is a pass-phrase. The Erlang's implementation expects that you provides actual key. I do not know how perl's Crypt::CBC derives the key from perspires. You can either reverse engineer that piece of code or you can use literal key. Best Regards, Dmitry > On Nov 10, 2015, at 11:58 AM, Bogdan Andu > wrote: > > use strict; > use warnings; > use MIME::Base64; > > use Crypt::CBC; > #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); > use PHP::Serialization qw(serialize unserialize); > > my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl 7bap 71district XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:"https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; > > print $pt, "\n"; > > my $key = "12345678900987654321001234567890"; > my $cipher = Crypt::CBC->new( > -key => $key, > -cipher => 'Blowfish', > -header => 'randomiv' > ); > > # print "$pt\n"; > my $encpt = $cipher->encrypt($pt); > print "\n$encpt", "\n"; > > print "\n", encode_base64($encpt), "\n"; > > ## TEST > my $decpt = $cipher->decrypt($encpt); > print "\n$decpt", "\n"; -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Nov 11 01:52:56 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 11 Nov 2015 13:52:56 +1300 Subject: [erlang-questions] EEP 45: New macros for function name and arity In-Reply-To: References: Message-ID: <11540C92-00CF-4CDA-8356-C6852FCBAC00@cs.otago.ac.nz> It's looking good, but there is something I do not understand. "The FUNCTION_NAME and FUNCTION_ARITY macros can be used in any form that starts with an atom followed by a left parenthesis." (1) Does this mean *before* or *after* macro expansion. -define(FOO, foo). ?FOO (?FOO) -> ?FOO. Is this a 'form that starts with an atom followed by a left parenthesis'? (2) The expansion of ?FUNCTION_NAME is an atom. So is *this* a 'form that starts with an atom followed by a left parenthesis'? ?FUNCTION_NAME() -> ?FUNCTION_ARITY. I think I have finally understood how ?FUNCTION_NAME and ?FUNCTION_ARITY can work in a function head. The trick is that when you first go through, you don't know _which_ atom and integer they are, but you do know _that_ they are an atom and an integer and cannot possibly be hiding any commas. So it can be done in two rounds. First a round of macro expansion as at present, but passing ?FUNCTION_NAME and ?FUNCTION_ARITY along as literals, then a round where they are replaced by their values. I am not suggesting that this is the way that this feature *should* be implemented, but that it took me a fair bit of mental effort to come up with a model for how this *could* work. And that suggests to me that something *like* this model should be part of the user documentation for this feature. The ?FUNCTION_STRING example is all very well, but as a bear of very little brain I could use a reminder of whether this is expected to work in a function head or not. Given that bar("test"++"/"++"3") -> foo. works it is certainly surprising that foo(atom_to_list(test)++"/"++integer_to_list(3)) -> bar. doesn't. This suggests to me that ?FUNCTION_STRING *won't* work in a function head, and the EEP should probably say so. I don't think there is any *need* for ?FUNCTION_STRING to work in a function head, it's just that since ?FUNCTION_NAME and ?FUNCTION_ARITY do, it deserves explicit mention. The FUNCTION_STRING example is followed by a section asking why FUNCTION_STRING doesn't exist. "To be able to reject invocations of FUNCTION in clauses" looks like an unfinished edit; "FUNCTION" should be "FUNCTION_NAME or FUNCTION_ARITY". Returning to FUNCTION_STRING, since bar(?MODULE_STRING) -> foo. works but foo(atom_to_list(?MODULE)) -> bar. does not, it appears that ?MODULE_STRING is not redundant, and that means that ?FUNCTION_STRING would not be redundant either. Mind you, using ?MODULE_STRING in a function head is much more plausible than using ?FUNCTION_STRING. Alternatively, the Erlang compiler could evaluate atom_to_list(Atom) and integer_to_list(Integer) in patterns... From ok@REDACTED Wed Nov 11 02:35:09 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 11 Nov 2015 14:35:09 +1300 Subject: [erlang-questions] Fourth draft of EEP 44 - Additional preprocessor directives In-Reply-To: References: <8018932F-D641-42F5-BB97-2F7CF7D017B4@cs.otago.ac.nz> Message-ID: <7BB96167-111F-49E7-BE0D-6C9DEAFFE049@cs.otago.ac.nz> On 6/11/2015, at 11:34 pm, Bj?rn Gustavsson wrote: > > That change to the language should be described > in a separate EEP. It can be implemented independently > of any changes to the preprocessor. It has. EEP 38. Written in 2011 and still no action. From bjorn@REDACTED Wed Nov 11 07:24:58 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Wed, 11 Nov 2015 07:24:58 +0100 Subject: [erlang-questions] EEP 45: New macros for function name and arity In-Reply-To: <11540C92-00CF-4CDA-8356-C6852FCBAC00@cs.otago.ac.nz> References: <11540C92-00CF-4CDA-8356-C6852FCBAC00@cs.otago.ac.nz> Message-ID: On Wed, Nov 11, 2015 at 1:52 AM, Richard A. O'Keefe wrote: > It's looking good, but there is something I do not > understand. > > "The FUNCTION_NAME and FUNCTION_ARITY macros can be used in any form that starts with an atom followed by a left parenthesis." > > (1) Does this mean *before* or *after* macro expansion. > -define(FOO, foo). > ?FOO (?FOO) -> ?FOO. Before macro expansion, as currently implemented. > Is this a 'form that starts with an atom followed > by a left parenthesis'? No. That means that the following example is illegal: -define(FOO, foo). ?FOO(?FOO) -> ?FUNCTION_NAME. The error message will be "?FUNCTION_NAME can only be used within a function". That is unfortunate. The preprocessor could do macro expansion before looking for an atom and a left parenthesis, but then we would have to think carefully about (2). > (2) The expansion of ?FUNCTION_NAME is an atom. > So is *this* a 'form that starts with an atom > followed by a left parenthesis'? > > ?FUNCTION_NAME() -> ?FUNCTION_ARITY. No. Currently, the error message would be "?FUNCTION_NAME can only be used within a function". Clearly, this example must be illegal, but the error message is not very good. I will have to think about this. > I think I have finally understood how ?FUNCTION_NAME > and ?FUNCTION_ARITY can work in a function head. The > trick is that when you first go through, you don't > know _which_ atom and integer they are, but you do > know _that_ they are an atom and an integer and cannot > possibly be hiding any commas. So it can be done in > two rounds. First a round of macro expansion as at > present, but passing ?FUNCTION_NAME and ?FUNCTION_ARITY > along as literals, then a round where they are replaced > by their values. Yes, that it is how I have implemented it. I only do two rounds if a macro invocation is actually found in the function head. [...] > The ?FUNCTION_STRING example is all very well, but as > a bear of very little brain I could use a reminder of > whether this is expected to work in a function head > or not. Given that > > bar("test"++"/"++"3") -> foo. > > works it is certainly surprising that > > foo(atom_to_list(test)++"/"++integer_to_list(3)) -> bar. > > doesn't. This suggests to me that ?FUNCTION_STRING > *won't* work in a function head, and the EEP should > probably say so. Correct. It won't. Operators are allowed in patterns, but not BIF calls. > I don't think there is any *need* for ?FUNCTION_STRING > to work in a function head, it's just that since > ?FUNCTION_NAME and ?FUNCTION_ARITY do, it deserves > explicit mention. OK. I'll mention it. > The FUNCTION_STRING example is followed by a section > asking why FUNCTION_STRING doesn't exist. > > "To be able to reject invocations of FUNCTION in clauses" > looks like an unfinished edit; "FUNCTION" should be > "FUNCTION_NAME or FUNCTION_ARITY". Yes, I missed that one. [...] > Alternatively, the Erlang compiler could evaluate > atom_to_list(Atom) and integer_to_list(Integer) > in patterns... > That would be the subject of a separate EEP. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From bog495@REDACTED Wed Nov 11 09:28:54 2015 From: bog495@REDACTED (Bogdan Andu) Date: Wed, 11 Nov 2015 10:28:54 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com> Message-ID: the purpose of this function was to handle encrypted data generated by Crypt::CBC perl module from cpan: http://search.cpan.org/~lds/Crypt-CBC-2.33/CBC.pm funny thing is there is php module (that emulates Crypt::CBC): http://pear.php.net/package/Crypt_CBC may be a more general approach would be to make use of PBKDF2 and *ALL* implementation must agree on a single pbkdf algorithm. I have also observed that padding is not handled also for blowfish_cbc block cipher, at least. Basically there are few things missing for non-ECB mode of operations (CBC, CFB, CRT, )for block ciphers, among them being padding and an universal password-based key derivation function algorithm. Bogdan On Wed, Nov 11, 2015 at 1:22 AM, Technion wrote: > Hi, > > > I'm not sure if you have other mitigating code, but I'd be concerned about > calling two calls to MD5 on a passphrase a "derived key" - this would be > very weak crypto. > > > ------------------------------ > *From:* erlang-questions-bounces@REDACTED < > erlang-questions-bounces@REDACTED> on behalf of Bogdan Andu < > bog495@REDACTED> > *Sent:* Wednesday, 11 November 2015 12:51 AM > *To:* Dmitry Kolesnikov > *Cc:* Erlang > *Subject:* Re: [erlang-questions] blowfish cbc mode decrypt > > I was thinking that is handled internally... > > however, this did the trick (for randomiv mode of operation): > > blowfish_key_from_key(Key) -> > Temp = crypto:md5(Key), > blowfish_key_from_key1(Temp). > > blowfish_key_from_key1(Temp) when size(Temp) < 56 -> > Temp1 = crypto:md5(Temp), > blowfish_key_from_key1(<>); > > blowfish_key_from_key1(Temp) -> > <> = Temp, > Temp1. > > I think that crypto should have this kind oh helper functions > because , like in cases of blowfish cipher , whose key length can > be variable, could be useful in using the cipher correctly, and people > would know that some things are not implied. > > Thanks, > Bogdan > > > On Tue, Nov 10, 2015 at 1:15 PM, Dmitry Kolesnikov > wrote: > >> Hello, >> >> Yes, this want I thought. You are using wrong key to decrypt data on >> Erlang side. >> >> Please notice that >> >> ?Crypt::CBC can derive the key and the IV from a passphrase that you >> provide, or can let you specify the true key and IV manually? >> >> The -key argument provides either a passphrase to use to generate the >> encryption key, or the literal value of the block cipher key. If used in >> passphrase mode (which is the default), -key can be any number of >> characters; the actual key will be derived by passing the passphrase >> through a series of MD5 hash operations.? >> >> So, in your example Key is not an encryption key, this is a pass-phrase. >> The Erlang?s implementation expects that you provides actual key. >> >> I do not know how perl?s Crypt::CBC derives the key from perspires. You >> can either reverse engineer that piece of code or you can use literal key. >> >> >> Best Regards, >> Dmitry >> >> >> > On Nov 10, 2015, at 11:58 AM, Bogdan Andu wrote: >> > >> > use strict; >> > use warnings; >> > use MIME::Base64; >> > >> > use Crypt::CBC; >> > #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); >> > use PHP::Serialization qw(serialize unserialize); >> > >> > my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST >> \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl >> 7bap 71district >> XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:" >> https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr >> 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 >> \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 >> \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 >> \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr >> 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; >> > >> > print $pt, "\n"; >> > >> > my $key = "12345678900987654321001234567890"; >> > my $cipher = Crypt::CBC->new( >> > -key => $key, >> > -cipher => 'Blowfish', >> > -header => 'randomiv' >> > ); >> > >> > # print "$pt\n"; >> > my $encpt = $cipher->encrypt($pt); >> > print "\n$encpt", "\n"; >> > >> > print "\n", encode_base64($encpt), "\n"; >> > >> > ## TEST >> > my $decpt = $cipher->decrypt($encpt); >> > print "\n$decpt", "\n"; >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anna.grzybowska@REDACTED Wed Nov 11 11:52:49 2015 From: anna.grzybowska@REDACTED (Anna Grzybowska) Date: Wed, 11 Nov 2015 10:52:49 +0000 Subject: [erlang-questions] ANN Berlin Elang Factory Lite Programme is Up! Message-ID: The Berlin Erlang Factory Lite programme is looking very interesting! On 1 Dec Berlin will be the best place in Europe to find out more about cool new Erlang and Elixir projects. Have a look at the programme here: http://www.erlang-factory.com/berlin2015#programme Check out our discounts and register here: http://www.erlang-factory.com/berlin2015#register See you in Berlin! -- Anna Grzybowska Conference Executive Erlang Solutions Ltd ?New Loom House?Back Church Lane?London? E1 1LU -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Wed Nov 11 15:06:44 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Wed, 11 Nov 2015 15:06:44 +0100 Subject: [erlang-questions] EEP 45: New macros for function name and arity Message-ID: This is the third draft. This draft will specify what will happen when the following code is compiled: -define(FOO, foo). ?FOO(?FOO) -> {?FUNCTION_NAME,?FUNCTION_ARITY}. ?FUNCTION_NAME() -> ?FUNCTION_ARITY. There is also additional justifications for why there is no ?FUNCTION_STRING macro. http://www.erlang.org/eeps/eep-0045.html https://github.com/erlang/eep/blob/master/eeps/eep-0045.md /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From ingela.andin@REDACTED Wed Nov 11 20:02:46 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 11 Nov 2015 20:02:46 +0100 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com> Message-ID: Hi! The public_key application has password based encryption functions, although they are currently not API functions we might consider making them API functions if there is an interest/need for it. Regards Ingela Erlang OTP -Team Ericsson AB 2015-11-11 9:28 GMT+01:00 Bogdan Andu : > the purpose of this function was to handle > encrypted data generated by Crypt::CBC perl module from cpan: > http://search.cpan.org/~lds/Crypt-CBC-2.33/CBC.pm > > funny thing is there is php module (that emulates Crypt::CBC): > http://pear.php.net/package/Crypt_CBC > > may be a more general approach would be to make use of PBKDF2 > and *ALL* implementation must agree on a single pbkdf algorithm. > > > I have also observed that padding is not handled also for blowfish_cbc > block cipher, at least. > > Basically there are few things missing for non-ECB mode of operations > (CBC, CFB, CRT, )for block ciphers, > among them being padding and an universal password-based key derivation > function algorithm. > > Bogdan > > On Wed, Nov 11, 2015 at 1:22 AM, Technion wrote: > >> Hi, >> >> >> I'm not sure if you have other mitigating code, but I'd be concerned >> about calling two calls to MD5 on a passphrase a "derived key" - this would >> be very weak crypto. >> >> >> ------------------------------ >> *From:* erlang-questions-bounces@REDACTED < >> erlang-questions-bounces@REDACTED> on behalf of Bogdan Andu < >> bog495@REDACTED> >> *Sent:* Wednesday, 11 November 2015 12:51 AM >> *To:* Dmitry Kolesnikov >> *Cc:* Erlang >> *Subject:* Re: [erlang-questions] blowfish cbc mode decrypt >> >> I was thinking that is handled internally... >> >> however, this did the trick (for randomiv mode of operation): >> >> blowfish_key_from_key(Key) -> >> Temp = crypto:md5(Key), >> blowfish_key_from_key1(Temp). >> >> blowfish_key_from_key1(Temp) when size(Temp) < 56 -> >> Temp1 = crypto:md5(Temp), >> blowfish_key_from_key1(<>); >> >> blowfish_key_from_key1(Temp) -> >> <> = Temp, >> Temp1. >> >> I think that crypto should have this kind oh helper functions >> because , like in cases of blowfish cipher , whose key length can >> be variable, could be useful in using the cipher correctly, and people >> would know that some things are not implied. >> >> Thanks, >> Bogdan >> >> >> On Tue, Nov 10, 2015 at 1:15 PM, Dmitry Kolesnikov < >> dmkolesnikov@REDACTED> wrote: >> >>> Hello, >>> >>> Yes, this want I thought. You are using wrong key to decrypt data on >>> Erlang side. >>> >>> Please notice that >>> >>> ?Crypt::CBC can derive the key and the IV from a passphrase that you >>> provide, or can let you specify the true key and IV manually? >>> >>> The -key argument provides either a passphrase to use to generate the >>> encryption key, or the literal value of the block cipher key. If used in >>> passphrase mode (which is the default), -key can be any number of >>> characters; the actual key will be derived by passing the passphrase >>> through a series of MD5 hash operations.? >>> >>> So, in your example Key is not an encryption key, this is a pass-phrase. >>> The Erlang?s implementation expects that you provides actual key. >>> >>> I do not know how perl?s Crypt::CBC derives the key from perspires. You >>> can either reverse engineer that piece of code or you can use literal key. >>> >>> >>> Best Regards, >>> Dmitry >>> >>> >>> > On Nov 10, 2015, at 11:58 AM, Bogdan Andu wrote: >>> > >>> > use strict; >>> > use warnings; >>> > use MIME::Base64; >>> > >>> > use Crypt::CBC; >>> > #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); >>> > use PHP::Serialization qw(serialize unserialize); >>> > >>> > my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST >>> \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl >>> 7bap 71district >>> XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:" >>> https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr >>> 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 >>> \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 >>> \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 >>> \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr >>> 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; >>> > >>> > print $pt, "\n"; >>> > >>> > my $key = "12345678900987654321001234567890"; >>> > my $cipher = Crypt::CBC->new( >>> > -key => $key, >>> > -cipher => 'Blowfish', >>> > -header => 'randomiv' >>> > ); >>> > >>> > # print "$pt\n"; >>> > my $encpt = $cipher->encrypt($pt); >>> > print "\n$encpt", "\n"; >>> > >>> > print "\n", encode_base64($encpt), "\n"; >>> > >>> > ## TEST >>> > my $decpt = $cipher->decrypt($encpt); >>> > print "\n$decpt", "\n"; >>> >>> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Nov 11 20:55:08 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 11 Nov 2015 22:55:08 +0300 Subject: [erlang-questions] driver_alloc high usage In-Reply-To: References: Message-ID: Just for a reminder: [case A of {driver_alloc,_} -> Calls = proplists:get_value(calls,I), {_,_,M} = lists:keyfind(driver_alloc,1,Calls),{_,_,F} = lists:keyfind(driver_free,1,Calls), io:format("~p ~p\n", [A, M-F]); _ -> ok end|| {A,I} <- recon_alloc:allocators()]. shows something like: {driver_alloc,0} 125581808 {driver_alloc,1} 313 {driver_alloc,2} 280 {driver_alloc,3} 131 {driver_alloc,4} -52 {driver_alloc,5} -152 ... Now need only to find out, who is mallocing without free -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Nov 11 21:48:34 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 11 Nov 2015 23:48:34 +0300 Subject: [erlang-questions] speed of neotoma parser Message-ID: Our Flussonic is using neotoma to parse our own config. Neotoma creates parser, this parser reads text config. It was great, users were really happy when we changed from erlang syntax to our own syntax, but now time to pay came. One user came with 280 KB config file (written by hands, yes). Neotoma parser parses it during 45 seconds. We've met with this problem earlier in our pulsedb library, when we changed very small (one screen) parser from neotoma to yecc and got 50-100 times speedup. Now it is not easy to do it, because our config parser takes about 1000 lines of peg file. I don't undertand what is the problem with speed of neotoma parser: 1) is it a nature of state machines that it generate? 2) is it a forgotten switch [{disable_slow,true}] in options? or is it something else? Maybe there is a simple thing that we can do with neotoma for it to generate faster parser? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Wed Nov 11 21:56:49 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 11 Nov 2015 22:56:49 +0200 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: Message-ID: Hello, I would suggest you to profile it with https://github.com/virtan/eep This is very good to tool to catch this type of things Best Regards, Dmitry > On 11 Nov 2015, at 22:48, Max Lapshin wrote: > > Our Flussonic is using neotoma to parse our own config. > > Neotoma creates parser, this parser reads text config. > > It was great, users were really happy when we changed from erlang syntax to our own syntax, but now time to pay came. > > One user came with 280 KB config file (written by hands, yes). Neotoma parser parses it during 45 seconds. > > We've met with this problem earlier in our pulsedb library, when we changed very small (one screen) parser from neotoma to yecc and got 50-100 times speedup. > > Now it is not easy to do it, because our config parser takes about 1000 lines of peg file. > > I don't undertand what is the problem with speed of neotoma parser: > 1) is it a nature of state machines that it generate? > 2) is it a forgotten switch [{disable_slow,true}] in options? > or is it something else? > > Maybe there is a simple thing that we can do with neotoma for it to generate faster parser? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ingela.andin@REDACTED Wed Nov 11 22:29:00 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 11 Nov 2015 22:29:00 +0100 Subject: [erlang-questions] Erlang SSL and Invalid Certificate Chains In-Reply-To: References: <1151586104.11805.1407832825589.JavaMail.zimbra@tpip.net> Message-ID: Hi! Sorry for the late reply, been really busy. Firstly I have not changed my mind about breaking specs, but actually the latest TLS draft is validating this behaviour and even recommending that it be handled so in earlier versions in defiance to the specs, so we are looking into including it after all. And yes doing it in the verify fun would mean to first pass all certificates and saving them in the user state, and then redo the validation with a new sorting of the chain, but you need not do the verification manually, you can call the same public_key API function that ssl does. It would work the only downside is that it all checks will be run twice. Regards Ingela Erlang/OTP team - Ericsson AB 2014-08-12 22:24 GMT+02:00 Ben Murphy : > Cheers, > > Do you know how I would go about writing the verify function? It looks > like it is called for every certificate in the chain from the last > certificate to the first certificate and I can either fail or pass at > each step. If I could know which cert was the 'first' one then I could > just succeed every certificate and then rebuild the chain when I reach > the first certificate and return pass or fail depending on whether i > thought it was valid. But it doesn't look like the verify function has > any concept of where it is in the chain relative to the first > certificate. I suspect the only safe way of doing it will be to always > return success in the verify function and build up the cert chain in > ets (or somewhere else) then after the ssl handshake is completed > manually validate the certificate chain. > > On Tue, Aug 12, 2014 at 7:21 PM, Ingela Andin > wrote: > > Hi! > > > > 2014-08-12 12:01 GMT+02:00 Ben Murphy : > > > >> I agree that these chains break the spec. The problem is almost > >> everyone accepts these chains. So if you don't accept these chains > >> then you are going to run into compatibility issues. I've tested > >> safari/firefox/chrome/openssl/golang/java and all these > >> implementations accept these chains. > >> > >> If you look at the implementations this is a deliberately coded feature: > >> > >> http://golang.org/src/pkg/crypto/x509/verify.go > >> func (c *Certificate) Verify(opts VerifyOptions) (chains > >> [][]*Certificate, err error) { > >> > >> I think most implementations treat the first certificate as the > >> clients certificate and the rest of the certificates as possible > >> intermediaries between the clients certificate and a trusted > >> certificate and try to build a chain based on that. > >> > >> > > > > Just because many people got it wrong does not make it right. > > If you want to interop with implementations that do not follow the spec, > you > > can do it with > > help of the verify fun. It might not be a compelling solution and it > might > > give some performance penalty, > > but it is doable. We do not desire to be bug-compatible with other > > implementations and make compromises that > > also might compromise with security. > > > > Regards Ingela Erlang/OTP team - Ericsson AB > > > > > > > >> > >> On Tue, Aug 12, 2014 at 9:40 AM, Andreas Schultz > >> wrote: > >> > Hi Ben, > >> > > >> > ----- Original Message ----- > >> >> Hi Ingela, > >> >> > >> >> So the real server I'm having trouble with is not public. But I have > a > >> >> made up server that is configured in a similar fashion. > >> >> > >> >> https://test.fonix.io/ > >> >> > >> >> I've just added an extra cert to the end of the chain. This validates > >> >> correctly in chrome and with curl on the command line. > >> >> > >> >> https://test.fonix.io:444/ > >> >> > >> >> Where this is an extra cert added to the end of the chain and the > >> >> order of the chain is broken. This validates correctly in chrome and > >> >> with curl on the command line. > >> >> > >> >> I think some browsers/ssl implementations may also accept chains > where > >> >> the first certificate in chain is not the client certificate but > nginx > >> >> won't let me test this scenario :) > >> > > >> > That would violate the standard. from RFC 2246 (TLS 1.0), Section > 7.4.2: > >> > > >> > certificate_list > >> > This is a sequence (chain) of X.509v3 certificates. The > sender's > >> > certificate must come first in the list. Each following > >> > certificate must directly certify the one preceding it. Because > >> > certificate validation requires that root keys be distributed > >> > independently, the self-signed certificate which specifies the > >> > root certificate authority may optionally be omitted from the > >> > chain, under the assumption that the remote end must already > >> > possess it in order to validate it in any case. > >> > > >> > RFC 5246 (TLS 1.2) adds this: > >> > > >> > The following rules apply to the certificates sent by the server: > >> > > >> > - The certificate type MUST be X.509v3, unless explicitly > negotiated > >> > otherwise (e.g., [TLSPGP]). > >> > > >> > - The end entity certificate's public key (and associated > >> > restrictions) MUST be compatible with the selected key exchange > >> > algorithm. > >> > > >> > So the ordering is strict, no unrelated certificates are permitted > >> > within the chain. This wording leaves it open whether a unrelated > >> > certificated is permitted at the end of the chain. However, I would > >> > rather choose the stricter interpretation and consider any unrelated > >> > certificate an error. > >> > > >> > The certificates on your server port 443 are: > >> > > >> > 0 > >> > > s:/serialNumber=oVMlbmYif-Pcg7kbO6LFKlAd0LjtU6Uw/C=GB/ST=London/L=London/O=Orca > >> > Digital Ltd/OU=IT/CN=*.fonix.io > >> > i:/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA > >> > 1 s:/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA > >> > i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA > >> > 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA > >> > i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority > >> > 3 s:/C=US/O=Google Inc/CN=Google Internet Authority G2 > >> > i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA > >> > > >> > The chain for Number 0, 1 and 2 seems to be ok. However the Number 3 > >> > certificate is obsolete and IMHO an error. > >> > > >> > > >> > The certificates on your server port 444 are: > >> > > >> > Certificate chain > >> > 0 > >> > > s:/serialNumber=oVMlbmYif-Pcg7kbO6LFKlAd0LjtU6Uw/C=GB/ST=London/L=London/O=Orca > >> > Digital Ltd/OU=IT/CN=*.fonix.io > >> > i:/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA > >> > 1 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA > >> > i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority > >> > 2 s:/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA > >> > i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA > >> > 3 s:/C=US/O=Google Inc/CN=Google Internet Authority G2 > >> > i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA > >> > > >> > So, Number 0 is ok, Number 1 is not the issuer of Number 0, so this IS > >> > invalid, Number 2 is the issuer of Number 0, and Number 3 is unrelated > >> > to any of them. > >> > > >> > To summarize: > >> > > >> > I believe Erlang is correct rejecting those certificate chains and > >> > Chrome and wget (or probably really OpenSSL) need to be fixed to > >> > properly reject those. > >> > > >> > Andreas > >> > > >> >> > >> >> I don't think I would be able to use verify_fun to safely ignore > >> >> arbitrary extra certificates while not accepting untrusted chains. I > >> >> believe I would need a whitelist of extra certificates to ignore > which > >> >> is problematic because things will explode when the server makes > >> >> changes that they think are 'safe'. > >> >> > >> >> On Mon, Aug 11, 2014 at 3:59 PM, Ingela Andin < > ingela.andin@REDACTED> > >> >> wrote: > >> >> > HI! > >> >> > > >> >> > 2014-08-11 16:16 GMT+02:00 Ben Murphy : > >> >> > > >> >> >> (I originally sent this to erlang-bugs but it doesn't look like it > >> >> >> made it there) > >> >> >> > >> >> >> We have a problem where an SSL server sends back a certificate > chain > >> >> >> that is invalid according to the TLS 1.2 specification and erlang > >> >> >> rejects this chain with an unknown ca error. However, openssl and > >> >> >> browsers will accept this chain because they are less strict about > >> >> >> validation. > >> >> >> > >> >> >> The chain looks something like: > >> >> >> > >> >> >> 0. Server Cert issued by Intermediate Cert > >> >> >> 1. Intermediate Cert issued by Root Cert > >> >> >> 2. Root Cert issued by Root Cert > >> >> >> 3. Unrelated certificate > >> >> >> 4. Unrelated certificate > >> >> > > >> >> > > >> >> > > >> >> > What server is sending such a chain? Could you give me a concrete > >> >> > example,maybe of list if you do not want to expose > >> >> > it to the world. > >> >> > > >> >> > > >> >> > > >> >> >> > >> >> >> > >> >> >> Which is invalid according to: > http://www.ietf.org/rfc/rfc5246.txt > >> >> >> > >> >> >> certificate_list > >> >> >> This is a sequence (chain) of certificates. The sender's > >> >> >> certificate MUST come first in the list. Each following > >> >> >> certificate MUST directly certify the one preceding it. > >> >> >> Because > >> >> >> certificate validation requires that root keys be > distributed > >> >> >> independently, the self-signed certificate that specifies > the > >> >> >> root > >> >> >> certificate authority MAY be omitted from the chain, under > the > >> >> >> assumption that the remote end must already possess it in > >> >> >> order to > >> >> >> validate it in any case > >> >> >> > >> >> >> Looking at the openssl code they start at the beginning of the > chain > >> >> >> then recursively find the issuer in order to build up a chain. > While > >> >> >> the erlang ssl code assumes the last certificate in the chain is > the > >> >> >> root CA (ssl_certificate:trusted_cert_and_path). > >> >> >> > >> >> > > >> >> > It assumes that the last certificate is the ROOT or the certificate > >> >> > just > >> >> > before > >> >> > the ROOT (as the ROOT cert may be left out as you said above) and > if > >> >> > the ROOT is left out it will try to find the ROOT in its database. > If > >> >> > the > >> >> > ROOT is part > >> >> > of the chain it will still be looked up to make sure we already > have > >> >> > it. > >> >> > > >> >> > > >> >> >> Maybe this is more of a feature request than a bug. But I was > >> >> >> wondering if it would be possible for erlang to either accept > these > >> >> >> dodgy chains, provide an option when connecting to accept these > >> >> >> dodgy > >> >> >> chains or allow users to supply a function to modify the > certificate > >> >> >> chain before validation takes place. > >> >> > > >> >> > > >> >> > > >> >> > You can use the verify_fun to accept any path validation errors you > >> >> > want, > >> >> > at your own risk as you are lowering the security requirements by > >> >> > doing so. > >> >> > > >> >> > [...] > >> >> > > >> >> > > >> >> > Regards Ingela Erlang/OTP team - Ericsson AB > >> >> _______________________________________________ > >> >> erlang-questions mailing list > >> >> erlang-questions@REDACTED > >> >> http://erlang.org/mailman/listinfo/erlang-questions > >> >> > >> > > >> > -- > >> > -- > >> > Dipl. Inform. > >> > Andreas Schultz > >> > > >> > email: as@REDACTED > >> > phone: +49-391-819099-224 > >> > mobil: +49-170-2226073 > >> > > >> > ------------------- enabling your networks ------------------- > >> > > >> > Travelping GmbH phone: +49-391-819099229 > >> > Roentgenstr. 13 fax: +49-391-819099299 > >> > D-39108 Magdeburg email: info@REDACTED > >> > GERMANY web: http://www.travelping.com > >> > > >> > Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 > >> > Geschaeftsfuehrer: Holger Winkelmann | VAT ID No.: DE236673780 > >> > -------------------------------------------------------------- > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Wed Nov 11 23:24:40 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Wed, 11 Nov 2015 22:24:40 +0000 Subject: [erlang-questions] driver_alloc high usage In-Reply-To: References: Message-ID: Since it is instance 0, it will most likely be someone that does not run in a scheduler thread. eg async thread or nif thread. On Wed, Nov 11, 2015 at 7:55 PM, Max Lapshin wrote: > Just for a reminder: > > [case A of {driver_alloc,_} -> Calls = proplists:get_value(calls,I), > {_,_,M} = lists:keyfind(driver_alloc,1,Calls),{_,_,F} = > lists:keyfind(driver_free,1,Calls), io:format("~p ~p\n", [A, M-F]); _ -> ok > end|| {A,I} <- recon_alloc:allocators()]. > > > shows something like: > > > {driver_alloc,0} 125581808 > > {driver_alloc,1} 313 > > {driver_alloc,2} 280 > > {driver_alloc,3} 131 > > {driver_alloc,4} -52 > > {driver_alloc,5} -152 > > ... > > > > Now need only to find out, who is mallocing without free > -------------- next part -------------- An HTML attachment was scrubbed... URL: From technion@REDACTED Thu Nov 12 00:00:50 2015 From: technion@REDACTED (Technion) Date: Wed, 11 Nov 2015 23:00:50 +0000 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com> , Message-ID: Hi, I appreciate in this case, the limitation is meeting compatibility with an existing product. In terms of "things missing", if the goal is just "strong crypto", it's hard to go past libsodium, which has this erlang binding: https://github.com/tonyg/erlang-nacl [https://avatars2.githubusercontent.com/u/17710?v=3&s=400] tonyg/erlang-nacl ? GitHub erlang-nacl - Erlang binding to NaCl in the form of libsodium. Read more... ________________________________ From: Bogdan Andu Sent: Wednesday, 11 November 2015 7:28 PM To: Technion Cc: Dmitry Kolesnikov; Erlang Subject: Re: [erlang-questions] blowfish cbc mode decrypt the purpose of this function was to handle encrypted data generated by Crypt::CBC perl module from cpan: http://search.cpan.org/~lds/Crypt-CBC-2.33/CBC.pm [http://www.gravatar.com/avatar/d5d94e49bade52d29be36e2a5259993b?r=g&s=80&d=http%3A%2F%2Fst.pimg.net%2Ftucs%2Fimg%2Fwho.png] Crypt::CBC - search.cpan.org NAME . Crypt::CBC - Encrypt Data with Cipher Block Chaining Mode. SYNOPSIS use Crypt::CBC; $cipher = Crypt::CBC->new( -key => 'my secret key ... Read more... funny thing is there is php module (that emulates Crypt::CBC): http://pear.php.net/package/Crypt_CBC may be a more general approach would be to make use of PBKDF2 and *ALL* implementation must agree on a single pbkdf algorithm. I have also observed that padding is not handled also for blowfish_cbc block cipher, at least. Basically there are few things missing for non-ECB mode of operations (CBC, CFB, CRT, )for block ciphers, among them being padding and an universal password-based key derivation function algorithm. Bogdan On Wed, Nov 11, 2015 at 1:22 AM, Technion > wrote: Hi, I'm not sure if you have other mitigating code, but I'd be concerned about calling two calls to MD5 on a passphrase a "derived key" - this would be very weak crypto. ________________________________ From: erlang-questions-bounces@REDACTED > on behalf of Bogdan Andu > Sent: Wednesday, 11 November 2015 12:51 AM To: Dmitry Kolesnikov Cc: Erlang Subject: Re: [erlang-questions] blowfish cbc mode decrypt I was thinking that is handled internally... however, this did the trick (for randomiv mode of operation): blowfish_key_from_key(Key) -> Temp = crypto:md5(Key), blowfish_key_from_key1(Temp). blowfish_key_from_key1(Temp) when size(Temp) < 56 -> Temp1 = crypto:md5(Temp), blowfish_key_from_key1(<>); blowfish_key_from_key1(Temp) -> <> = Temp, Temp1. I think that crypto should have this kind oh helper functions because , like in cases of blowfish cipher , whose key length can be variable, could be useful in using the cipher correctly, and people would know that some things are not implied. Thanks, Bogdan On Tue, Nov 10, 2015 at 1:15 PM, Dmitry Kolesnikov > wrote: Hello, Yes, this want I thought. You are using wrong key to decrypt data on Erlang side. Please notice that ?Crypt::CBC can derive the key and the IV from a passphrase that you provide, or can let you specify the true key and IV manually? The -key argument provides either a passphrase to use to generate the encryption key, or the literal value of the block cipher key. If used in passphrase mode (which is the default), -key can be any number of characters; the actual key will be derived by passing the passphrase through a series of MD5 hash operations.? So, in your example Key is not an encryption key, this is a pass-phrase. The Erlang?s implementation expects that you provides actual key. I do not know how perl?s Crypt::CBC derives the key from perspires. You can either reverse engineer that piece of code or you can use literal key. Best Regards, Dmitry > On Nov 10, 2015, at 11:58 AM, Bogdan Andu > wrote: > > use strict; > use warnings; > use MIME::Base64; > > use Crypt::CBC; > #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); > use PHP::Serialization qw(serialize unserialize); > > my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl 7bap 71district XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:"https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; > > print $pt, "\n"; > > my $key = "12345678900987654321001234567890"; > my $cipher = Crypt::CBC->new( > -key => $key, > -cipher => 'Blowfish', > -header => 'randomiv' > ); > > # print "$pt\n"; > my $encpt = $cipher->encrypt($pt); > print "\n$encpt", "\n"; > > print "\n", encode_base64($encpt), "\n"; > > ## TEST > my $decpt = $cipher->decrypt($encpt); > print "\n$decpt", "\n"; -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Nov 12 04:04:09 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 12 Nov 2015 16:04:09 +1300 Subject: [erlang-questions] EEP 45: New macros for function name and arity In-Reply-To: References: <11540C92-00CF-4CDA-8356-C6852FCBAC00@cs.otago.ac.nz> Message-ID: <566847BC-59B1-4284-9237-F0B3F7AC3917@cs.otago.ac.nz> The current draft * defines FUNCTION_STRING (in Examples) * has a section saying why FUNCTION_STRING doesn't exist. I suggest that the section title should be "Why do I have to define FUNCTION_STRING myself?" and that the most important reason is that there are *two* reasonable definitions -define(FUNCTION_STRING, atom_to_list(?FUNCTION_NAME) ++ "/" ++ integer_to_list(?FUNCTION_ARITY)). and -define(FUNCTION_STRING, ?MODULE_STRING ++ ":" ++ atom_to_list(?FUNCTION_NAME) ++ "/" ++ integer_to_list(?FUNCTION_ARITY)). and that it's up to you to pick the one you need. That section starts with There already exist MODULE and MODULE_STRING macros, so why no MODULE_STRING macro? ^^^^^^^^^^^^^ where the second MODULE_STRING should be FUNCTION_STRING. It would be nice if the Examples section included self_ref(Data, Handler) -> ... Handler(fun ?FUNCTION_NAME/?FUNCTION_ARITY, Data) ... Now the message I am replying to says that -define(FOO, foo). ?FOO(?FOO) -> ?FOO. is *not* a 'form that starts with an atom followed by a left parenthesis'. But the EEP specifically says The following is also legal: -define(A, a). ?A(?FUNCTION_NAME) -> ok. Perhaps the simplest model is just to say that ?FUNCTION_NAME (and of course ?FUNCTION_ARITY) may not begin any form. From bjorn@REDACTED Thu Nov 12 07:35:10 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Thu, 12 Nov 2015 07:35:10 +0100 Subject: [erlang-questions] EEP 45: New macros for function name and arity In-Reply-To: <566847BC-59B1-4284-9237-F0B3F7AC3917@cs.otago.ac.nz> References: <11540C92-00CF-4CDA-8356-C6852FCBAC00@cs.otago.ac.nz> <566847BC-59B1-4284-9237-F0B3F7AC3917@cs.otago.ac.nz> Message-ID: Thanks for the corrections and suggestions. I will incorporate them into the next draft which I will probably prepare at the beginning of next week. /Bj?rn On Thu, Nov 12, 2015 at 4:04 AM, Richard A. O'Keefe wrote: > The current draft > > * defines FUNCTION_STRING (in Examples) > > * has a section saying why FUNCTION_STRING > doesn't exist. > > I suggest that the section title should be > "Why do I have to define FUNCTION_STRING myself?" > and that the most important reason is that there > are *two* reasonable definitions > > -define(FUNCTION_STRING, > atom_to_list(?FUNCTION_NAME) ++ "/" ++ > integer_to_list(?FUNCTION_ARITY)). > > and > > -define(FUNCTION_STRING, > ?MODULE_STRING ++ ":" ++ > atom_to_list(?FUNCTION_NAME) ++ "/" ++ > integer_to_list(?FUNCTION_ARITY)). > > and that it's up to you to pick the one you need. > That section starts with > > There already exist MODULE and MODULE_STRING macros, > so why no MODULE_STRING macro? > ^^^^^^^^^^^^^ > > where the second MODULE_STRING should be FUNCTION_STRING. > > It would be nice if the Examples section included > > self_ref(Data, Handler) -> > ... > Handler(fun ?FUNCTION_NAME/?FUNCTION_ARITY, Data) > ... > > Now the message I am replying to says that > -define(FOO, foo). > ?FOO(?FOO) -> ?FOO. > is *not* a 'form that starts with an atom followed > by a left parenthesis'. But the EEP specifically > says > The following is also legal: > > -define(A, a). > ?A(?FUNCTION_NAME) -> ok. > > Perhaps the simplest model is just to say that > ?FUNCTION_NAME (and of course ?FUNCTION_ARITY) > may not begin any form. > > > > > > -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From max.lapshin@REDACTED Thu Nov 12 08:04:08 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 12 Nov 2015 10:04:08 +0300 Subject: [erlang-questions] driver_alloc high usage In-Reply-To: References: Message-ID: And as far as I understand leak is somewhere in driver code, not c library. Btw, maybe it is possible to pass some port id to driver_alloc for measuring, what instance is leaking? Driver_alloc(size, port_instance_id) On Thursday, November 12, 2015, Lukas Larsson wrote: > Since it is instance 0, it will most likely be someone that does not run > in a scheduler thread. eg async thread or nif thread. > > On Wed, Nov 11, 2015 at 7:55 PM, Max Lapshin > wrote: > >> Just for a reminder: >> >> [case A of {driver_alloc,_} -> Calls = proplists:get_value(calls,I), >> {_,_,M} = lists:keyfind(driver_alloc,1,Calls),{_,_,F} = >> lists:keyfind(driver_free,1,Calls), io:format("~p ~p\n", [A, M-F]); _ -> ok >> end|| {A,I} <- recon_alloc:allocators()]. >> >> >> shows something like: >> >> >> {driver_alloc,0} 125581808 >> >> {driver_alloc,1} 313 >> >> {driver_alloc,2} 280 >> >> {driver_alloc,3} 131 >> >> {driver_alloc,4} -52 >> >> {driver_alloc,5} -152 >> >> ... >> >> >> >> Now need only to find out, who is mallocing without free >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.french@REDACTED Thu Nov 12 07:23:26 2015 From: michael.french@REDACTED (French, Michael) Date: Thu, 12 Nov 2015 06:23:26 +0000 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: , Message-ID: <55A810C00B7CDA48A5AB43C566064B41014C52D889@SE-EX019.groupinfra.com> PEG packrat parser generators in Erlang have a lot of scope for performance problems: - text character input, should use binary for large input - random access in lists, should use tuples for small input - non-memoized performance problem, multiple execution of same parse - memoized memory problem, possibly multiple copies of input - slow random character access in UTF8 binary, needs btree index? You could also try: https://github.com/joearms/elib1/tree/master/unsupported/epeg but it uses a different PEG grammar, different embedded Erlang rule binding, and it's 'unsupported'. Does anyone have any performance comparison of epeg and neotoma? BR Mike ________________________________________ From: erlang-questions-bounces@REDACTED [erlang-questions-bounces@REDACTED] on behalf of Dmitry Kolesnikov [dmkolesnikov@REDACTED] Sent: Wednesday, November 11, 2015 11:56 PM To: Max Lapshin Cc: Erlang-Questions Questions Subject: Re: [erlang-questions] speed of neotoma parser Hello, I would suggest you to profile it with https://github.com/virtan/eep This is very good to tool to catch this type of things Best Regards, Dmitry > On 11 Nov 2015, at 22:48, Max Lapshin wrote: > > Our Flussonic is using neotoma to parse our own config. > > Neotoma creates parser, this parser reads text config. > > It was great, users were really happy when we changed from erlang syntax to our own syntax, but now time to pay came. > > One user came with 280 KB config file (written by hands, yes). Neotoma parser parses it during 45 seconds. > > We've met with this problem earlier in our pulsedb library, when we changed very small (one screen) parser from neotoma to yecc and got 50-100 times speedup. > > Now it is not easy to do it, because our config parser takes about 1000 lines of peg file. > > I don't undertand what is the problem with speed of neotoma parser: > 1) is it a nature of state machines that it generate? > 2) is it a forgotten switch [{disable_slow,true}] in options? > or is it something else? > > Maybe there is a simple thing that we can do with neotoma for it to generate faster parser? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From bog495@REDACTED Thu Nov 12 10:03:49 2015 From: bog495@REDACTED (Bogdan Andu) Date: Thu, 12 Nov 2015 11:03:49 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com> Message-ID: RFC2898 (PKCS#5) describes PBKDF2 function, so would be nice to have such functions available for Key derivation for encryption and for MAC computations At least this could be handled in user code. CouchDB project has also a PBKDF2 implementation Other issues would be : 1) padding for CBC mode - should be done internally 2) an universal accepted (and implemented) protocol for key derivation in encryption schemes Bogdan On Thu, Nov 12, 2015 at 1:00 AM, Technion wrote: > Hi, > > > I appreciate in this case, the limitation is meeting compatibility with an > existing product. > > > In terms of "things missing", if the goal is just "strong crypto", it's > hard to go past libsodium, which has this erlang binding: > > > https://github.com/tonyg/erlang-nacl > > > tonyg/erlang-nacl ? GitHub > erlang-nacl - Erlang binding to NaCl in the form of libsodium. > Read more... > > > > > > > ------------------------------ > *From:* Bogdan Andu > *Sent:* Wednesday, 11 November 2015 7:28 PM > *To:* Technion > *Cc:* Dmitry Kolesnikov; Erlang > *Subject:* Re: [erlang-questions] blowfish cbc mode decrypt > > the purpose of this function was to handle > encrypted data generated by Crypt::CBC perl module from cpan: > http://search.cpan.org/~lds/Crypt-CBC-2.33/CBC.pm > > > Crypt::CBC - search.cpan.org > NAME . Crypt::CBC - Encrypt Data with Cipher Block Chaining Mode. SYNOPSIS > use Crypt::CBC; $cipher = Crypt::CBC->new( -key => 'my secret key ... > Read more... > > > > > funny thing is there is php module (that emulates Crypt::CBC): > http://pear.php.net/package/Crypt_CBC > > may be a more general approach would be to make use of PBKDF2 > and *ALL* implementation must agree on a single pbkdf algorithm. > > > I have also observed that padding is not handled also for blowfish_cbc > block cipher, at least. > > Basically there are few things missing for non-ECB mode of operations > (CBC, CFB, CRT, )for block ciphers, > among them being padding and an universal password-based key derivation > function algorithm. > > Bogdan > > On Wed, Nov 11, 2015 at 1:22 AM, Technion wrote: > >> Hi, >> >> >> I'm not sure if you have other mitigating code, but I'd be concerned >> about calling two calls to MD5 on a passphrase a "derived key" - this would >> be very weak crypto. >> >> >> ------------------------------ >> *From:* erlang-questions-bounces@REDACTED < >> erlang-questions-bounces@REDACTED> on behalf of Bogdan Andu < >> bog495@REDACTED> >> *Sent:* Wednesday, 11 November 2015 12:51 AM >> *To:* Dmitry Kolesnikov >> *Cc:* Erlang >> *Subject:* Re: [erlang-questions] blowfish cbc mode decrypt >> >> I was thinking that is handled internally... >> >> however, this did the trick (for randomiv mode of operation): >> >> blowfish_key_from_key(Key) -> >> Temp = crypto:md5(Key), >> blowfish_key_from_key1(Temp). >> >> blowfish_key_from_key1(Temp) when size(Temp) < 56 -> >> Temp1 = crypto:md5(Temp), >> blowfish_key_from_key1(<>); >> >> blowfish_key_from_key1(Temp) -> >> <> = Temp, >> Temp1. >> >> I think that crypto should have this kind oh helper functions >> because , like in cases of blowfish cipher , whose key length can >> be variable, could be useful in using the cipher correctly, and people >> would know that some things are not implied. >> >> Thanks, >> Bogdan >> >> >> On Tue, Nov 10, 2015 at 1:15 PM, Dmitry Kolesnikov < >> dmkolesnikov@REDACTED> wrote: >> >>> Hello, >>> >>> Yes, this want I thought. You are using wrong key to decrypt data on >>> Erlang side. >>> >>> Please notice that >>> >>> ?Crypt::CBC can derive the key and the IV from a passphrase that you >>> provide, or can let you specify the true key and IV manually? >>> >>> The -key argument provides either a passphrase to use to generate the >>> encryption key, or the literal value of the block cipher key. If used in >>> passphrase mode (which is the default), -key can be any number of >>> characters; the actual key will be derived by passing the passphrase >>> through a series of MD5 hash operations.? >>> >>> So, in your example Key is not an encryption key, this is a pass-phrase. >>> The Erlang?s implementation expects that you provides actual key. >>> >>> I do not know how perl?s Crypt::CBC derives the key from perspires. You >>> can either reverse engineer that piece of code or you can use literal key. >>> >>> >>> Best Regards, >>> Dmitry >>> >>> >>> > On Nov 10, 2015, at 11:58 AM, Bogdan Andu wrote: >>> > >>> > use strict; >>> > use warnings; >>> > use MIME::Base64; >>> > >>> > use Crypt::CBC; >>> > #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); >>> > use PHP::Serialization qw(serialize unserialize); >>> > >>> > my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST >>> \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl >>> 7bap 71district >>> XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:" >>> https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr >>> 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 >>> \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 >>> \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 >>> \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr >>> 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; >>> > >>> > print $pt, "\n"; >>> > >>> > my $key = "12345678900987654321001234567890"; >>> > my $cipher = Crypt::CBC->new( >>> > -key => $key, >>> > -cipher => 'Blowfish', >>> > -header => 'randomiv' >>> > ); >>> > >>> > # print "$pt\n"; >>> > my $encpt = $cipher->encrypt($pt); >>> > print "\n$encpt", "\n"; >>> > >>> > print "\n", encode_base64($encpt), "\n"; >>> > >>> > ## TEST >>> > my $decpt = $cipher->decrypt($encpt); >>> > print "\n$decpt", "\n"; >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Thu Nov 12 10:22:42 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 12 Nov 2015 11:22:42 +0200 Subject: [erlang-questions] blowfish cbc mode decrypt In-Reply-To: References: <7113E121-A27E-415E-8364-062F79B6CBD1@gmail.com> Message-ID: Hello, Meanwhile, you can use this one: https://github.com/fogfish/feta/blob/master/src/hash.erl#L177 Best Regards, Dmitry > On Nov 12, 2015, at 11:03 AM, Bogdan Andu wrote: > > RFC2898 (PKCS#5) describes PBKDF2 function, > so would be nice to have such functions available > for Key derivation for encryption and for MAC computations > At least this could be handled in user code. > > CouchDB project has also a PBKDF2 implementation > > Other issues would be : > 1) padding for CBC mode - should be done internally > 2) an universal accepted (and implemented) protocol > for key derivation in encryption schemes > > Bogdan > > On Thu, Nov 12, 2015 at 1:00 AM, Technion wrote: > Hi, > > > I appreciate in this case, the limitation is meeting compatibility with an existing product. > > > In terms of "things missing", if the goal is just "strong crypto", it's hard to go past libsodium, which has this erlang binding: > > > https://github.com/tonyg/erlang-nacl > > > > tonyg/erlang-nacl ? GitHub > erlang-nacl - Erlang binding to NaCl in the form of libsodium. > Read more... > > > > > > > From: Bogdan Andu > Sent: Wednesday, 11 November 2015 7:28 PM > To: Technion > Cc: Dmitry Kolesnikov; Erlang > Subject: Re: [erlang-questions] blowfish cbc mode decrypt > > the purpose of this function was to handle > encrypted data generated by Crypt::CBC perl module from cpan: > http://search.cpan.org/~lds/Crypt-CBC-2.33/CBC.pm > > > Crypt::CBC - search.cpan.org > NAME . Crypt::CBC - Encrypt Data with Cipher Block Chaining Mode. SYNOPSIS use Crypt::CBC; $cipher = Crypt::CBC->new( -key => 'my secret key ... > Read more... > > > > > funny thing is there is php module (that emulates Crypt::CBC): > http://pear.php.net/package/Crypt_CBC > > may be a more general approach would be to make use of PBKDF2 > and *ALL* implementation must agree on a single pbkdf algorithm. > > > I have also observed that padding is not handled also for blowfish_cbc > block cipher, at least. > > Basically there are few things missing for non-ECB mode of operations (CBC, CFB, CRT, )for block ciphers, > among them being padding and an universal password-based key derivation function algorithm. > > Bogdan > > On Wed, Nov 11, 2015 at 1:22 AM, Technion wrote: > Hi, > > > I'm not sure if you have other mitigating code, but I'd be concerned about calling two calls to MD5 on a passphrase a "derived key" - this would be very weak crypto. > > > From: erlang-questions-bounces@REDACTED on behalf of Bogdan Andu > Sent: Wednesday, 11 November 2015 12:51 AM > To: Dmitry Kolesnikov > Cc: Erlang > Subject: Re: [erlang-questions] blowfish cbc mode decrypt > > I was thinking that is handled internally... > > however, this did the trick (for randomiv mode of operation): > > blowfish_key_from_key(Key) -> > Temp = crypto:md5(Key), > blowfish_key_from_key1(Temp). > > blowfish_key_from_key1(Temp) when size(Temp) < 56 -> > Temp1 = crypto:md5(Temp), > blowfish_key_from_key1(<>); > > blowfish_key_from_key1(Temp) -> > <> = Temp, > Temp1. > > I think that crypto should have this kind oh helper functions > because , like in cases of blowfish cipher , whose key length can > be variable, could be useful in using the cipher correctly, and people > would know that some things are not implied. > > Thanks, > Bogdan > > > On Tue, Nov 10, 2015 at 1:15 PM, Dmitry Kolesnikov wrote: > Hello, > > Yes, this want I thought. You are using wrong key to decrypt data on Erlang side. > > Please notice that > > ?Crypt::CBC can derive the key and the IV from a passphrase that you provide, or can let you specify the true key and IV manually? > > The -key argument provides either a passphrase to use to generate the encryption key, or the literal value of the block cipher key. If used in passphrase mode (which is the default), -key can be any number of characters; the actual key will be derived by passing the passphrase through a series of MD5 hash operations.? > > So, in your example Key is not an encryption key, this is a pass-phrase. The Erlang?s implementation expects that you provides actual key. > > I do not know how perl?s Crypt::CBC derives the key from perspires. You can either reverse engineer that piece of code or you can use literal key. > > > Best Regards, > Dmitry > > > > On Nov 10, 2015, at 11:58 AM, Bogdan Andu wrote: > > > > use strict; > > use warnings; > > use MIME::Base64; > > > > use Crypt::CBC; > > #use Digest::HMAC_SHA1 qw(hmac_sha1 hmac_sha1_hex); > > use PHP::Serialization qw(serialize unserialize); > > > > my $pt = 'a:10:{s:6:"adresa";s:89:"Address 2 TEST \xc3\x84\xc2\x83\xc3\x83\xc2\xae\xc3\x88\xc2\x99\xc3\x88\xc2\x9b\xc3\x88\xc2\x99\xc3\x88\xc2\x9bbl 7bap 71district XXXBucure\xc3\x88\xc2\x99tiJUDE\xc3\x88\xc2\x9a031905RO";s:4:"info";i:1460382;s:7:"urlback";s:41:"https://192.162.16.116:8020/snep_response";s:4:"cuip";s:18:"Cererea nr 1460382";s:6:"idtaxa";i:5001;s:5:"email";s:16:"xxx123@REDACTED";s:4:"nume";s:55:"\xc3\x88\xc2\x99 \xc3\x88\xc2\x9b \xc3\x84\xc2\x83 \xc3\x83\xc2\xae \xc3\x83\xc2\xa2 \xc3\x83\xc2\x82 \xc3\x83\xc2\x8e \xc3\x84\xc2\x82 \xc3\x88\xc2\x98 \xc3\x88\xc2\x9a u\xc3\x83\xc2\xa7";s:3:"cui";s:18:"Cererea nr 1460382";s:9:"idnomunic";i:13;s:4:"suma";d:262.69;}'; > > > > print $pt, "\n"; > > > > my $key = "12345678900987654321001234567890"; > > my $cipher = Crypt::CBC->new( > > -key => $key, > > -cipher => 'Blowfish', > > -header => 'randomiv' > > ); > > > > # print "$pt\n"; > > my $encpt = $cipher->encrypt($pt); > > print "\n$encpt", "\n"; > > > > print "\n", encode_base64($encpt), "\n"; > > > > ## TEST > > my $decpt = $cipher->decrypt($encpt); > > print "\n$decpt", "\n"; > > > > From jesper.louis.andersen@REDACTED Thu Nov 12 15:53:53 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 12 Nov 2015 15:53:53 +0100 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: Message-ID: On Wed, Nov 11, 2015 at 9:48 PM, Max Lapshin wrote: > We've met with this problem earlier in our pulsedb library, when we > changed very small (one screen) parser from neotoma to yecc and got 50-100 > times speedup. Based on your speedup here, I'd be willing to bet there is something inside neotoma that is * rather easy to optimize * yields good speedup Two orders of magnitude is simply so far away in parsing speed (for a simpler grammar system) that I'm willing to bet you are doing some work you don't need to do. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Thu Nov 12 17:06:03 2015 From: rvirding@REDACTED (Robert Virding) Date: Thu, 12 Nov 2015 17:06:03 +0100 Subject: [erlang-questions] Languages running on BEAM In-Reply-To: <5640B775.1090700@llaisdy.com> References: <5640B775.1090700@llaisdy.com> Message-ID: I would point out that Eric Merrit was very wrong about aspects of LFE in his comparison with Joxa which makes some parts of it not very meaningful. Robert On 9 November 2015 at 16:10, Ivan Uemlianin wrote: > Dear All > > I am compiling a list of programming languages that run on BEAM, the > Erlang VM: > > https://github.com/llaisdy/beam_languages > > The resource is aimed at people who might be thinking of implementing such > a language themselves. Consequently the list includes experimental and > abandoned languages as well as the stars. > > I am also including a section listing documents (papers, presentations, > blog posts, etc.) that cover the theme of implementing languages for the > BEAM in general. > > Languages listed so far: > > - efene > - elixir > - erl2 > - erlang > - erlog > - erlyjs > - haskerl > - joxa > - lfe > - luerl > - reia > - 'the concurrent schemer' > - xerl > > If you know about a language or a document that should be on one of these > lists --- or if I've made mistakes! ---, please let me know (email, > issue/PR, tweet, etc.). > > With thanks and best wishes > > Ivan > > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > @llaisdy > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > festina lente > ============================================================ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Nov 12 18:07:15 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 12 Nov 2015 20:07:15 +0300 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: Message-ID: Yes, Louis, I also think that there may be a simple way of speeding it up. I'm only afraid that I will have to open my university book and remember what LL-1 means and how it differs from LALR =) Ok, will try to profile it first. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Nov 12 18:08:12 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 12 Nov 2015 20:08:12 +0300 Subject: [erlang-questions] driver_alloc high usage In-Reply-To: References: Message-ID: You were exactly right. It was outdated erlang-sqlite3 driver that leaked from an async thread. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Thu Nov 12 17:14:36 2015 From: ivan@REDACTED (Ivan Uemlianin) Date: Thu, 12 Nov 2015 16:14:36 +0000 Subject: [erlang-questions] Languages running on BEAM In-Reply-To: References: <5640B775.1090700@llaisdy.com> Message-ID: <5644BAEC.9090609@llaisdy.com> Dear Robert Thanks for your comment. I think an objective comparison (i.e. by a fourth party) of the three different erlang lisps will definitely be in order. Best wishes Ivan On 12/11/2015 16:06, Robert Virding wrote: > I would point out that Eric Merrit was very wrong about aspects of LFE > in his comparison with Joxa which makes some parts of it not very > meaningful. > > Robert > > > On 9 November 2015 at 16:10, Ivan Uemlianin > wrote: > > Dear All > > I am compiling a list of programming languages that run on BEAM, > the Erlang VM: > > https://github.com/llaisdy/beam_languages > > The resource is aimed at people who might be thinking of > implementing such a language themselves. Consequently the list > includes experimental and abandoned languages as well as the stars. > > I am also including a section listing documents (papers, > presentations, blog posts, etc.) that cover the theme of > implementing languages for the BEAM in general. > > Languages listed so far: > > - efene > - elixir > - erl2 > - erlang > - erlog > - erlyjs > - haskerl > - joxa > - lfe > - luerl > - reia > - 'the concurrent schemer' > - xerl > > If you know about a language or a document that should be on one > of these lists --- or if I've made mistakes! ---, please let me > know (email, issue/PR, tweet, etc.). > > With thanks and best wishes > > Ivan > > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > @llaisdy > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > > festina lente > ============================================================ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED @llaisdy llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nyarly@REDACTED Thu Nov 12 19:08:43 2015 From: nyarly@REDACTED (Judson Lester) Date: Thu, 12 Nov 2015 18:08:43 +0000 Subject: [erlang-questions] Make sure epmd is running? Message-ID: I just ran across this code I wrote a few months ago: _Hack = os:cmd("epmd -daemon"), % XXX dirty hack to make sure epmd is running Obviously, I promised myself at the time that I'd fix it later. Is there a better fix, though? The program is a command line app, so ideally I'd rather not require that the operator know about epmd (or indeed anything about operating an Erlang system, if possible), so I considered something about "here's how to make sure that epmd starts at boot" or something, but it seemed worse given the use case than the above. So, is there a tidy way to start-epmd-if-not-running from within Erlang? Judson -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Nov 12 19:45:04 2015 From: g@REDACTED (Garrett Smith) Date: Thu, 12 Nov 2015 12:45:04 -0600 Subject: [erlang-questions] Make sure epmd is running? In-Reply-To: References: Message-ID: I would treat this as a requirement for Erlang in the first place (it is certainly for distributed Erlang) and orchestrate epmd start/monitoring at the OS level. For dev mode on your workstation, the hack is probably fine. For a production system, start epmd explicitly and supervise it (e.g. using systemd, runit, etc. - whatever you're system is using for init). On Thu, Nov 12, 2015 at 12:08 PM, Judson Lester wrote: > I just ran across this code I wrote a few months ago: > > _Hack = os:cmd("epmd -daemon"), % XXX dirty hack to make sure epmd is > running > > Obviously, I promised myself at the time that I'd fix it later. Is there a > better fix, though? The program is a command line app, so ideally I'd rather > not require that the operator know about epmd (or indeed anything about > operating an Erlang system, if possible), so I considered something about > "here's how to make sure that epmd starts at boot" or something, but it > seemed worse given the use case than the above. > > So, is there a tidy way to start-epmd-if-not-running from within Erlang? > > Judson > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From nyarly@REDACTED Thu Nov 12 19:48:25 2015 From: nyarly@REDACTED (Judson Lester) Date: Thu, 12 Nov 2015 18:48:25 +0000 Subject: [erlang-questions] Decomposing Modules Message-ID: I suspect I'm not alone in having modules that grow to unwieldy sizes. I'm starting to understand better general classes of module that make sense to address, but I'm wondering if there's advice about particular features within a module that suggest how to separate functions into smaller, more module chunks. My slowly developing intuition is that it has a lot to do with the parameters of the functions, and I get the sense that there's a gap in my understanding that'll make the decomposition easier. Maybe a refactoring of those parameters that'll make the separation more obvious? Judson -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Nov 12 21:05:11 2015 From: g@REDACTED (Garrett Smith) Date: Thu, 12 Nov 2015 14:05:11 -0600 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: What's an example of a really large module that is triggering your spidey senses? On Thu, Nov 12, 2015 at 12:48 PM, Judson Lester wrote: > I suspect I'm not alone in having modules that grow to unwieldy sizes. I'm > starting to understand better general classes of module that make sense to > address, but I'm wondering if there's advice about particular features > within a module that suggest how to separate functions into smaller, more > module chunks. > > My slowly developing intuition is that it has a lot to do with the > parameters of the functions, and I get the sense that there's a gap in my > understanding that'll make the decomposition easier. Maybe a refactoring of > those parameters that'll make the separation more obvious? > > Judson > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From nyarly@REDACTED Thu Nov 12 22:53:37 2015 From: nyarly@REDACTED (Judson Lester) Date: Thu, 12 Nov 2015 21:53:37 +0000 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: It's the depgraph manager for a build tool. The module is >1000 lines, with 280 function clauses, and shares some type specs and record defs through a .hrl. The project isn't public yet, but maybe I should get over that. On Thu, Nov 12, 2015 at 12:05 PM Garrett Smith wrote: > What's an example of a really large module that is triggering your > spidey senses? > > On Thu, Nov 12, 2015 at 12:48 PM, Judson Lester wrote: > > I suspect I'm not alone in having modules that grow to unwieldy sizes. > I'm > > starting to understand better general classes of module that make sense > to > > address, but I'm wondering if there's advice about particular features > > within a module that suggest how to separate functions into smaller, more > > module chunks. > > > > My slowly developing intuition is that it has a lot to do with the > > parameters of the functions, and I get the sense that there's a gap in my > > understanding that'll make the decomposition easier. Maybe a refactoring > of > > those parameters that'll make the separation more obvious? > > > > Judson > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nyarly@REDACTED Thu Nov 12 22:58:10 2015 From: nyarly@REDACTED (Judson Lester) Date: Thu, 12 Nov 2015 21:58:10 +0000 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: Yeah, so, for my sins: https://git.lrdesign.com/judson/ergo/blob/master/src/ergo_graphs.erl On Thu, Nov 12, 2015 at 1:53 PM Judson Lester wrote: > It's the depgraph manager for a build tool. The module is >1000 lines, > with 280 function clauses, and shares some type specs and record defs > through a .hrl. The project isn't public yet, but maybe I should get over > that. > > On Thu, Nov 12, 2015 at 12:05 PM Garrett Smith wrote: > >> What's an example of a really large module that is triggering your >> spidey senses? >> >> On Thu, Nov 12, 2015 at 12:48 PM, Judson Lester wrote: >> > I suspect I'm not alone in having modules that grow to unwieldy sizes. >> I'm >> > starting to understand better general classes of module that make sense >> to >> > address, but I'm wondering if there's advice about particular features >> > within a module that suggest how to separate functions into smaller, >> more >> > module chunks. >> > >> > My slowly developing intuition is that it has a lot to do with the >> > parameters of the functions, and I get the sense that there's a gap in >> my >> > understanding that'll make the decomposition easier. Maybe a >> refactoring of >> > those parameters that'll make the separation more obvious? >> > >> > Judson >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Nov 12 23:03:56 2015 From: g@REDACTED (Garrett Smith) Date: Thu, 12 Nov 2015 16:03:56 -0600 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: Without spending much time on this, my first impression is that you may want to decouple the gen_server implementation from the data structure implementation. These are probably separate concerns. The gen_server deals with state initialization and mutation for processes. The data structure implementation is a bunch of pure functions. On Thu, Nov 12, 2015 at 3:58 PM, Judson Lester wrote: > Yeah, so, for my sins: > https://git.lrdesign.com/judson/ergo/blob/master/src/ergo_graphs.erl > > On Thu, Nov 12, 2015 at 1:53 PM Judson Lester wrote: >> >> It's the depgraph manager for a build tool. The module is >1000 lines, >> with 280 function clauses, and shares some type specs and record defs >> through a .hrl. The project isn't public yet, but maybe I should get over >> that. >> >> On Thu, Nov 12, 2015 at 12:05 PM Garrett Smith wrote: >>> >>> What's an example of a really large module that is triggering your >>> spidey senses? >>> >>> On Thu, Nov 12, 2015 at 12:48 PM, Judson Lester wrote: >>> > I suspect I'm not alone in having modules that grow to unwieldy sizes. >>> > I'm >>> > starting to understand better general classes of module that make sense >>> > to >>> > address, but I'm wondering if there's advice about particular features >>> > within a module that suggest how to separate functions into smaller, >>> > more >>> > module chunks. >>> > >>> > My slowly developing intuition is that it has a lot to do with the >>> > parameters of the functions, and I get the sense that there's a gap in >>> > my >>> > understanding that'll make the decomposition easier. Maybe a >>> > refactoring of >>> > those parameters that'll make the separation more obvious? >>> > >>> > Judson >>> > >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> > From nyarly@REDACTED Thu Nov 12 23:16:08 2015 From: nyarly@REDACTED (Judson Lester) Date: Thu, 12 Nov 2015 22:16:08 +0000 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: The further question, in my mind, is: does it make sense to split out the queries from the data structure? Or some of the utility stuff that transform representations of edges to and from statements for instance. On Thu, Nov 12, 2015 at 2:04 PM Garrett Smith wrote: > Without spending much time on this, my first impression is that you > may want to decouple the gen_server implementation from the data > structure implementation. These are probably separate concerns. > > The gen_server deals with state initialization and mutation for > processes. The data structure implementation is a bunch of pure > functions. > > On Thu, Nov 12, 2015 at 3:58 PM, Judson Lester wrote: > > Yeah, so, for my sins: > > https://git.lrdesign.com/judson/ergo/blob/master/src/ergo_graphs.erl > > > > On Thu, Nov 12, 2015 at 1:53 PM Judson Lester wrote: > >> > >> It's the depgraph manager for a build tool. The module is >1000 lines, > >> with 280 function clauses, and shares some type specs and record defs > >> through a .hrl. The project isn't public yet, but maybe I should get > over > >> that. > >> > >> On Thu, Nov 12, 2015 at 12:05 PM Garrett Smith wrote: > >>> > >>> What's an example of a really large module that is triggering your > >>> spidey senses? > >>> > >>> On Thu, Nov 12, 2015 at 12:48 PM, Judson Lester > wrote: > >>> > I suspect I'm not alone in having modules that grow to unwieldy > sizes. > >>> > I'm > >>> > starting to understand better general classes of module that make > sense > >>> > to > >>> > address, but I'm wondering if there's advice about particular > features > >>> > within a module that suggest how to separate functions into smaller, > >>> > more > >>> > module chunks. > >>> > > >>> > My slowly developing intuition is that it has a lot to do with the > >>> > parameters of the functions, and I get the sense that there's a gap > in > >>> > my > >>> > understanding that'll make the decomposition easier. Maybe a > >>> > refactoring of > >>> > those parameters that'll make the separation more obvious? > >>> > > >>> > Judson > >>> > > >>> > _______________________________________________ > >>> > erlang-questions mailing list > >>> > erlang-questions@REDACTED > >>> > http://erlang.org/mailman/listinfo/erlang-questions > >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Nov 12 23:55:34 2015 From: g@REDACTED (Garrett Smith) Date: Thu, 12 Nov 2015 16:55:34 -0600 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: Could be :) If a set of functions stand out as forming a coherent interface for some common scenarios, that might be a candidate for a separate module. On the other hand, some modules are natural sinks for functions - e.g. erlang or an application 'util' module. For me, there's a clear separation of concerns between process management (e.g. gen_server) and data type manipulation (e.g. dict, graphs, etc.) Other concerns might deal with IO for a subsystem (e.g. database, service, etc.) I wouldn't really worry too much about this and let real pain drive your refactoring. There's no rule that says a module must be small or limited to N functions. There was even a proposal by a prominent Erlang programmer to get rid of modules altogether and just have a single global namespace for functions. You could experiment with various interfaces and see if that makes any difference to your understanding, usability, or maintainability of the software you're building. I would time box that though as you run the risk of over thinking this problem of keeping modules small. It might not be an actual problem. On Thu, Nov 12, 2015 at 4:16 PM, Judson Lester wrote: > The further question, in my mind, is: does it make sense to split out the > queries from the data structure? Or some of the utility stuff that transform > representations of edges to and from statements for instance. > > On Thu, Nov 12, 2015 at 2:04 PM Garrett Smith wrote: >> >> Without spending much time on this, my first impression is that you >> may want to decouple the gen_server implementation from the data >> structure implementation. These are probably separate concerns. >> >> The gen_server deals with state initialization and mutation for >> processes. The data structure implementation is a bunch of pure >> functions. >> >> On Thu, Nov 12, 2015 at 3:58 PM, Judson Lester wrote: >> > Yeah, so, for my sins: >> > https://git.lrdesign.com/judson/ergo/blob/master/src/ergo_graphs.erl >> > >> > On Thu, Nov 12, 2015 at 1:53 PM Judson Lester wrote: >> >> >> >> It's the depgraph manager for a build tool. The module is >1000 lines, >> >> with 280 function clauses, and shares some type specs and record defs >> >> through a .hrl. The project isn't public yet, but maybe I should get >> >> over >> >> that. >> >> >> >> On Thu, Nov 12, 2015 at 12:05 PM Garrett Smith wrote: >> >>> >> >>> What's an example of a really large module that is triggering your >> >>> spidey senses? >> >>> >> >>> On Thu, Nov 12, 2015 at 12:48 PM, Judson Lester >> >>> wrote: >> >>> > I suspect I'm not alone in having modules that grow to unwieldy >> >>> > sizes. >> >>> > I'm >> >>> > starting to understand better general classes of module that make >> >>> > sense >> >>> > to >> >>> > address, but I'm wondering if there's advice about particular >> >>> > features >> >>> > within a module that suggest how to separate functions into smaller, >> >>> > more >> >>> > module chunks. >> >>> > >> >>> > My slowly developing intuition is that it has a lot to do with the >> >>> > parameters of the functions, and I get the sense that there's a gap >> >>> > in >> >>> > my >> >>> > understanding that'll make the decomposition easier. Maybe a >> >>> > refactoring of >> >>> > those parameters that'll make the separation more obvious? >> >>> > >> >>> > Judson >> >>> > >> >>> > _______________________________________________ >> >>> > erlang-questions mailing list >> >>> > erlang-questions@REDACTED >> >>> > http://erlang.org/mailman/listinfo/erlang-questions >> >>> > From g@REDACTED Fri Nov 13 00:04:40 2015 From: g@REDACTED (Garrett Smith) Date: Thu, 12 Nov 2015 17:04:40 -0600 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: Could be :) If a set of functions stand out as forming a coherent interface for some common scenarios, that might be a candidate for a separate module. On the other hand, some modules are natural sinks for functions - e.g. erlang or an application 'util' module. For me, there's a clear separation of concerns between process management (e.g. gen_server) and data type manipulation (e.g. dict, graphs, etc.) Other concerns might deal with IO for a subsystem (e.g. database, service, etc.) Still others with a type of computation, etc. I wouldn't really worry too much about this and let real pain drive your refactoring. There's no rule that says a module must be small or limited to N functions. There was even a proposal by a prominent Erlang programmer to get rid of modules altogether and just have a single global namespace for functions. You could experiment with various interfaces and see if that makes any difference to your understanding, usability, or maintainability of the software you're building. I would time-box that though as you run the risk of over thinking this problem of keeping modules small. It might not be an actual problem. On Thu, Nov 12, 2015 at 4:16 PM, Judson Lester wrote: > The further question, in my mind, is: does it make sense to split out the > queries from the data structure? Or some of the utility stuff that transform > representations of edges to and from statements for instance. > > On Thu, Nov 12, 2015 at 2:04 PM Garrett Smith wrote: >> >> Without spending much time on this, my first impression is that you >> may want to decouple the gen_server implementation from the data >> structure implementation. These are probably separate concerns. >> >> The gen_server deals with state initialization and mutation for >> processes. The data structure implementation is a bunch of pure >> functions. >> >> On Thu, Nov 12, 2015 at 3:58 PM, Judson Lester wrote: >> > Yeah, so, for my sins: >> > https://git.lrdesign.com/judson/ergo/blob/master/src/ergo_graphs.erl >> > >> > On Thu, Nov 12, 2015 at 1:53 PM Judson Lester wrote: >> >> >> >> It's the depgraph manager for a build tool. The module is >1000 lines, >> >> with 280 function clauses, and shares some type specs and record defs >> >> through a .hrl. The project isn't public yet, but maybe I should get >> >> over >> >> that. >> >> >> >> On Thu, Nov 12, 2015 at 12:05 PM Garrett Smith wrote: >> >>> >> >>> What's an example of a really large module that is triggering your >> >>> spidey senses? >> >>> >> >>> On Thu, Nov 12, 2015 at 12:48 PM, Judson Lester >> >>> wrote: >> >>> > I suspect I'm not alone in having modules that grow to unwieldy >> >>> > sizes. >> >>> > I'm >> >>> > starting to understand better general classes of module that make >> >>> > sense >> >>> > to >> >>> > address, but I'm wondering if there's advice about particular >> >>> > features >> >>> > within a module that suggest how to separate functions into smaller, >> >>> > more >> >>> > module chunks. >> >>> > >> >>> > My slowly developing intuition is that it has a lot to do with the >> >>> > parameters of the functions, and I get the sense that there's a gap >> >>> > in >> >>> > my >> >>> > understanding that'll make the decomposition easier. Maybe a >> >>> > refactoring of >> >>> > those parameters that'll make the separation more obvious? >> >>> > >> >>> > Judson >> >>> > >> >>> > _______________________________________________ >> >>> > erlang-questions mailing list >> >>> > erlang-questions@REDACTED >> >>> > http://erlang.org/mailman/listinfo/erlang-questions >> >>> > From wallentin.dahlberg@REDACTED Fri Nov 13 00:09:15 2015 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Fri, 13 Nov 2015 00:09:15 +0100 Subject: [erlang-questions] [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: References: <5640D376.9010409@ericsson.com> <5644C00D.4020407@ericsson.com> <5644D84D.1060101@ericsson.com> Message-ID: If it's https://github.com/iguberman/otp/commit/1968f7207e8f80e77d0c2c4e3b50d018cdf8359b you are nearly there. You must prefix your function with the module name, in this case 'binary_'. i.e. BIF_RETTYPE *binary_*free_refc_bin_1(BIF_ALIST_1){ ... } Aside: Your in dangerous water with your BIF but I guess you know that. // Bj?rn-Egil 2015-11-12 20:01 GMT+01:00 Irina Guberman : > Hello dear Erlang developers, > > I would tremendously appreciate any help or if anyone can point me to > someone who can help with linking Erlang/OTP. I created an experimental > BIF in my own fork of erlang/otp, but can't get past linking stage (on Mac: Undefined > symbols for architecture x86_64, but same thing happens on ubuntu 14.04) > even though I checked everywhere and my BIF seems to be everywhere an > existing BIF from the same module is (with same naming conventions and > all). I don't know where to post this question either (Erlang Central > didn't work which is understandable, it's not like people are creating BIFs > everyday), I'm so desperate I'm about to post it on Twitter ;) > > Thanks a million for reading this message and even more so for pointing me > to where I can get help with this! > > Irina. > > > > > On Thu, Nov 12, 2015 at 12:19 PM, Hans Nilsson R < > hans.r.nilsson@REDACTED> wrote: > >> Great! If there also is documentation and a test case the likelihood of >> acceptance is higher :) >> -Hans >> >> On 11/12/2015 06:31 PM, Vipin Nair wrote: >> > Hello Hans, >> > >> >> A more general way to pass options to the key callback module would be >> >> to introduce a second form of the key_cb option: >> >> - keep todays {key_cb, Module::atom()} to keep compatibility >> >> - introduce the variant {key_cb, {Module::atom(), ModuleOptions}} >> >> [...] >> >> In that way it is possible to write a callback module with whatever is >> >> needed as private options. >> > >> > This will work! In retrospect, I should have done this instead. This is >> much >> > simpler and solves my problem. I'll send a new pull request with the >> suggested >> > implementation. Thanks! >> > >> >> >> _______________________________________________ >> erlang-patches mailing list >> erlang-patches@REDACTED >> http://erlang.org/mailman/listinfo/erlang-patches >> >> > > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://erlang.org/mailman/listinfo/erlang-patches > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nyarly@REDACTED Fri Nov 13 00:15:44 2015 From: nyarly@REDACTED (Judson Lester) Date: Thu, 12 Nov 2015 23:15:44 +0000 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: On Thu, Nov 12, 2015 at 2:55 PM Garrett Smith wrote: > Could be :) > > If a set of functions stand out as forming a coherent interface for > some common scenarios, that might be a candidate for a separate > module. On the other hand, some modules are natural sinks for > functions - e.g. erlang or an application 'util' module. > > For me, there's a clear separation of concerns between process > management (e.g. gen_server) and data type manipulation (e.g. dict, > graphs, etc.) Other concerns might deal with IO for a subsystem (e.g. > database, service, etc.) > > I wouldn't really worry too much about this and let real pain drive > your refactoring. There's no rule that says a module must be small or > limited to N functions. There was even a proposal by a prominent > Erlang programmer to get rid of modules altogether and just have a > single global namespace for functions. > > You could experiment with various interfaces and see if that makes any > difference to your understanding, usability, or maintainability of the > software you're building. I would time box that though as you run the > risk of over thinking this problem of keeping modules small. It might > not be an actual problem. > > To be completely honest, what's happening is that I am feeling the pain, but having a hard time seeing where to do the separation. I see the *concerns* but not necessarily how to map that into "this one goes here, that one goes there," but I guess the best answer may be "just try something and see" -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Fri Nov 13 00:44:56 2015 From: g@REDACTED (Garrett Smith) Date: Thu, 12 Nov 2015 17:44:56 -0600 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: I don't mind the scope of the module that you have there. It's longish, but not unreadable. I'd start by peeling apart the process concerns (state lifecycle and interactions) with data structure related concerns. Data structure here is not btw process state, but graph related operations. That might be easier said than done in looking at the code - it's not sure obvious where process ops and end and the others begin. I'd take that step though for sure as it would clarify the concerns there quiet a bit and should be fun. Life might go much easier with that one step. On Thu, Nov 12, 2015 at 5:15 PM, Judson Lester wrote: > > On Thu, Nov 12, 2015 at 2:55 PM Garrett Smith wrote: >> >> Could be :) >> >> If a set of functions stand out as forming a coherent interface for >> some common scenarios, that might be a candidate for a separate >> module. On the other hand, some modules are natural sinks for >> functions - e.g. erlang or an application 'util' module. >> >> For me, there's a clear separation of concerns between process >> management (e.g. gen_server) and data type manipulation (e.g. dict, >> graphs, etc.) Other concerns might deal with IO for a subsystem (e.g. >> database, service, etc.) >> >> I wouldn't really worry too much about this and let real pain drive >> your refactoring. There's no rule that says a module must be small or >> limited to N functions. There was even a proposal by a prominent >> Erlang programmer to get rid of modules altogether and just have a >> single global namespace for functions. >> >> You could experiment with various interfaces and see if that makes any >> difference to your understanding, usability, or maintainability of the >> software you're building. I would time box that though as you run the >> risk of over thinking this problem of keeping modules small. It might >> not be an actual problem. >> > > To be completely honest, what's happening is that I am feeling the pain, but > having a hard time seeing where to do the separation. I see the *concerns* > but not necessarily how to map that into "this one goes here, that one goes > there," but I guess the best answer may be "just try something and see" From zxq9@REDACTED Fri Nov 13 01:09:49 2015 From: zxq9@REDACTED (zxq9) Date: Fri, 13 Nov 2015 09:09:49 +0900 Subject: [erlang-questions] Decomposing Modules In-Reply-To: References: Message-ID: <3082237.ovZKYsoZ4q@changa> On 2015?11?12? ??? 18:48:25 Judson Lester wrote: > I suspect I'm not alone in having modules that grow to unwieldy sizes. I'm > starting to understand better general classes of module that make sense to > address, but I'm wondering if there's advice about particular features > within a module that suggest how to separate functions into smaller, more > module chunks. > > My slowly developing intuition is that it has a lot to do with the > parameters of the functions, and I get the sense that there's a gap in my > understanding that'll make the decomposition easier. Maybe a refactoring of > those parameters that'll make the separation more obvious? I would first ask "why is this a problem?" Big modules are not inherently bad, but covoluted modules can be very hard to read and maintain. One thing that sticks out to me is the mixing of semantics. The business language is mixed with the language of math and the language of the program's mechanics. You can't avoid mixing program mechanics in there somewhere; its got to run and be callable. But consider this: -export([statement_for_edge/1, edge_for_statement/1, remove_statement/2]). The first two are pure functions that probably don't belong here. They seem to be more general utilities (otherwise why export them?). The last one is peculiar, as it is exported from this module, not called internally, takes your #state{} record as an argument, and calls ets:match/2 on a table owned by...? So that's sort of odd. Maybe that one doesn't belong here, either. Your gen_server's state should stay with your gen_server. There is also a remove_statements/3 function that (at first glance) does not appear to be a plural form of remove_statement/2. Even if it effectively is a plural form of remove_statement/2, remove_statements operates in a very different way, wrapping change_graph/4 in nicer semantics. It appears that you've implemented a graph manipulation utility -- and very nearly a tiny graph database. It seems likely that the details of graph manipulation are a *lot* more general than their specific application in this product module (or rather what appears to be a product information (task?) management module -- there is no explanation of how this module fits into the system, so I'm left guessing). The module name is 'ergo_graphs' I might consider making it actually be *just* a graphing module, and create a separate 'ergo_product_manager_thingy' module that does all the product specific stuff that relates to the broader application. This difference between the general manipulation of graphs and the specific way in which they happen to be used within your program may be why you are beginning to get a creeping sense that "refactoring of those parameters will make the separation more obvious". It seems likely that the graph-bits and the gen_server bits can be separated in a useful way. Blah blah. These are details. My point is that there are signs that: 1. There are a likely few universal utilities mixed into the module. 2. The gen_server is not always being treated as a service. 3. The module is simply *going to be large* as long as it deals both with the business/application semantics and the details of how that translates to more general graph functions. If this thing works *right now* then there is no rush -- but whatever you can factor out of it would very likely become more generally useful in other ways (maybe in other projects) later. -Craig From zandra.hird@REDACTED Fri Nov 13 11:27:10 2015 From: zandra.hird@REDACTED (Zandra Hird) Date: Fri, 13 Nov 2015 11:27:10 +0100 Subject: [erlang-questions] Patch Package OTP 18.1.4 Released Message-ID: <5645BAFE.2010604@ericsson.com> Patch Package: OTP 18.1.4 Git Tag: OTP-18.1.4 Date: 2015-11-13 Trouble Report Id: OTP-13061, OTP-13062 Seq num: System: OTP Release: 18 Application: inets-6.0.3 Predecessor: OTP 18.1.3 Check out the git tag OTP-18.1.4, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- POTENTIAL INCOMPATIBILITIES ------------------------------------- --------------------------------------------------------------------- OTP-13062 Application(s): inets Add possibility to set socket options, such as nodelay, for httpd. Also phase out legacy option value inet6bf4 for the ipfamily option. This value will be translated to the value inet. --------------------------------------------------------------------- --- inets-6.0.3 ----------------------------------------------------- --------------------------------------------------------------------- The inets-6.0.3 application can be applied independently of other applications on a full OTP 18 installation. --- Fixed Bugs and Malfunctions --- OTP-13061 Application(s): inets Improved error handling and gracfully termination when an invalid chunked length header is encountered. --- Improvements and New Features --- OTP-13062 Application(s): inets *** POTENTIAL INCOMPATIBILITY *** Add possibility to set socket options, such as nodelay, for httpd. Also phase out legacy option value inet6bf4 for the ipfamily option. This value will be translated to the value inet. Full runtime dependencies of inets-6.0.3: erts-6.0, kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, ssl-5.3.4, stdlib-2.0 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From seancribbs@REDACTED Fri Nov 13 15:02:53 2015 From: seancribbs@REDACTED (Sean Cribbs) Date: Fri, 13 Nov 2015 09:02:53 -0500 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: Message-ID: Max, Do you have a link to your grammar? I can probably poke at it and give you some tips. However, I am well aware of performance problems with neotoma -- with large grammars or large inputs it drags. Yes, there are general problems for PEGs in Erlang, but its current implementation is particularly naive and wasteful. I'm working on a rewrite, but it's a complete overhaul (and more faithful to the thesis and reference implementation "Pappy"). Since it's not core to my day-job, I've only been able to work on the rewrite occasionally in my free time. On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin wrote: > Yes, Louis, I also think that there may be a simple way of speeding it up. > > I'm only afraid that I will have to open my university book and remember > what LL-1 means and how it differs from LALR =) > > Ok, will try to profile it first. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Fri Nov 13 21:25:22 2015 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 13 Nov 2015 21:25:22 +0100 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: Message-ID: PEG parsers are notoriously inefficient. How about having a separate tokenization pass, and parse token instead of characters. At a guess this would be far faster since you'd backtrack over completed tokens rather than characters. /Joe On Fri, Nov 13, 2015 at 3:02 PM, Sean Cribbs wrote: > Max, > > Do you have a link to your grammar? I can probably poke at it and give you > some tips. > > However, I am well aware of performance problems with neotoma -- with large > grammars or large inputs it drags. Yes, there are general problems for PEGs > in Erlang, but its current implementation is particularly naive and > wasteful. I'm working on a rewrite, but it's a complete overhaul (and more > faithful to the thesis and reference implementation "Pappy"). Since it's not > core to my day-job, I've only been able to work on the rewrite occasionally > in my free time. > > On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin wrote: >> >> Yes, Louis, I also think that there may be a simple way of speeding it up. >> >> I'm only afraid that I will have to open my university book and remember >> what LL-1 means and how it differs from LALR =) >> >> Ok, will try to profile it first. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From michael.french@REDACTED Sat Nov 14 14:05:26 2015 From: michael.french@REDACTED (French, Michael) Date: Sat, 14 Nov 2015 13:05:26 +0000 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: , Message-ID: <55A810C00B7CDA48A5AB43C566064B41014C52EAD0@SE-EX019.groupinfra.com> The concept of 'token' is fluid in PEGs. The terminal/non-terminal distinction might not work. For example, the definition of 'alphanumeric' might appear in several different 'tokens', rather than repeating char classes, which means the token rules become non-terminals. Maybe use a hint in the grammar, like using an upper-case name for rule LHS to indicate a 'token' (which is similar to a convention in Antlr). Then always memo-ize the token rules, but not (necessarily) the others that have lower-case rule names. BR Mike ________________________________________ From: erlang-questions-bounces@REDACTED [erlang-questions-bounces@REDACTED] on behalf of Joe Armstrong [erlang@REDACTED] Sent: Friday, November 13, 2015 11:25 PM To: Sean Cribbs Cc: Erlang-Questions Questions Subject: Re: [erlang-questions] speed of neotoma parser PEG parsers are notoriously inefficient. How about having a separate tokenization pass, and parse token instead of characters. At a guess this would be far faster since you'd backtrack over completed tokens rather than characters. /Joe On Fri, Nov 13, 2015 at 3:02 PM, Sean Cribbs wrote: > Max, > > Do you have a link to your grammar? I can probably poke at it and give you > some tips. > > However, I am well aware of performance problems with neotoma -- with large > grammars or large inputs it drags. Yes, there are general problems for PEGs > in Erlang, but its current implementation is particularly naive and > wasteful. I'm working on a rewrite, but it's a complete overhaul (and more > faithful to the thesis and reference implementation "Pappy"). Since it's not > core to my day-job, I've only been able to work on the rewrite occasionally > in my free time. > > On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin wrote: >> >> Yes, Louis, I also think that there may be a simple way of speeding it up. >> >> I'm only afraid that I will have to open my university book and remember >> what LL-1 means and how it differs from LALR =) >> >> Ok, will try to profile it first. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From max.lapshin@REDACTED Sat Nov 14 16:27:00 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 14 Nov 2015 18:27:00 +0300 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: <55A810C00B7CDA48A5AB43C566064B41014C52EAD0@SE-EX019.groupinfra.com> References: <55A810C00B7CDA48A5AB43C566064B41014C52EAD0@SE-EX019.groupinfra.com> Message-ID: I've sent example from production to Sean. (flussonic@REDACTED)2> timer:tc(fun() -> config2_format:parse(element(2,file:read_file("big.conf"))), ok end). {29727926,ok} On Sat, Nov 14, 2015 at 4:05 PM, French, Michael wrote: > The concept of 'token' is fluid in PEGs. The terminal/non-terminal > distinction might not work. For example, the definition of 'alphanumeric' > might appear in several different 'tokens', rather than repeating char > classes, which means the token rules become non-terminals. > > Maybe use a hint in the grammar, like using an upper-case name for rule > LHS to indicate a 'token' (which is similar to a convention in Antlr). Then > always memo-ize the token rules, but not (necessarily) the others that have > lower-case rule names. > > BR > Mike > > ________________________________________ > From: erlang-questions-bounces@REDACTED [ > erlang-questions-bounces@REDACTED] on behalf of Joe Armstrong [ > erlang@REDACTED] > Sent: Friday, November 13, 2015 11:25 PM > To: Sean Cribbs > Cc: Erlang-Questions Questions > Subject: Re: [erlang-questions] speed of neotoma parser > > PEG parsers are notoriously inefficient. How about having a separate > tokenization pass, and parse token instead of characters. At a guess > this would be far faster since you'd backtrack over completed tokens > rather than characters. > > /Joe > > On Fri, Nov 13, 2015 at 3:02 PM, Sean Cribbs wrote: > > Max, > > > > Do you have a link to your grammar? I can probably poke at it and give > you > > some tips. > > > > However, I am well aware of performance problems with neotoma -- with > large > > grammars or large inputs it drags. Yes, there are general problems for > PEGs > > in Erlang, but its current implementation is particularly naive and > > wasteful. I'm working on a rewrite, but it's a complete overhaul (and > more > > faithful to the thesis and reference implementation "Pappy"). Since it's > not > > core to my day-job, I've only been able to work on the rewrite > occasionally > > in my free time. > > > > On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin > wrote: > >> > >> Yes, Louis, I also think that there may be a simple way of speeding it > up. > >> > >> I'm only afraid that I will have to open my university book and remember > >> what LL-1 means and how it differs from LALR =) > >> > >> Ok, will try to profile it first. > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > >> > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From seancribbs@REDACTED Sat Nov 14 18:20:04 2015 From: seancribbs@REDACTED (Sean Cribbs) Date: Sat, 14 Nov 2015 12:20:04 -0500 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: <55A810C00B7CDA48A5AB43C566064B41014C52EAD0@SE-EX019.groupinfra.com> Message-ID: Hi Max, My general impression of your grammar is that it conflates syntax and semantics. Your configuration lines seem to be structured roughly like KEYWORD VALUES* ( "{" BLOCK "}" )? ";", but you create a single rule for every possible configuration item. This means at the top-level you have to create ordered choices between completely unrelated extents. Every config item you parse is going to go through the same backtracking among ~30 alternatives, with nothing able to be memoized (because the stem of each one is unique). This is a quintessential worst-case for the types of parsers that neotoma currently generates. I suggest you rethink the parser, focus on what are the shared structures and try to extract those. Use fewer rules, try not to treat each config item as a special case, and then validate the structure after parsing it. Alternatively, you could pick up and use cuttlefish which has a much simpler configuration language and built-in semantic validation and transformation and command-line tools. As you say earlier in the thread, you could also use leex and yecc. I understand the desire not to, but their performance characteristics are better known and more predictable. They might be worth it. I feel pretty responsible for the limitations you encountered. Ford's thesis clearly outlines optimizations that neotoma does not do, including more efficient matching of terminals via "switch", cost calculation to determine what to memoize and inline, unrolling recognition logic instead of using parser-combinators, and more. I'm working on a rewrite, but it's not ready yet. I'm sorry. I hope the suggestions above help you focus the grammar into something more usable. Sean On Sat, Nov 14, 2015 at 10:27 AM, Max Lapshin wrote: > I've sent example from production to Sean. > > (flussonic@REDACTED)2> timer:tc(fun() -> > config2_format:parse(element(2,file:read_file("big.conf"))), ok end). > > {29727926,ok} > > On Sat, Nov 14, 2015 at 4:05 PM, French, Michael > wrote: > >> The concept of 'token' is fluid in PEGs. The terminal/non-terminal >> distinction might not work. For example, the definition of 'alphanumeric' >> might appear in several different 'tokens', rather than repeating char >> classes, which means the token rules become non-terminals. >> >> Maybe use a hint in the grammar, like using an upper-case name for rule >> LHS to indicate a 'token' (which is similar to a convention in Antlr). Then >> always memo-ize the token rules, but not (necessarily) the others that have >> lower-case rule names. >> >> BR >> Mike >> >> ________________________________________ >> From: erlang-questions-bounces@REDACTED [ >> erlang-questions-bounces@REDACTED] on behalf of Joe Armstrong [ >> erlang@REDACTED] >> Sent: Friday, November 13, 2015 11:25 PM >> To: Sean Cribbs >> Cc: Erlang-Questions Questions >> Subject: Re: [erlang-questions] speed of neotoma parser >> >> PEG parsers are notoriously inefficient. How about having a separate >> tokenization pass, and parse token instead of characters. At a guess >> this would be far faster since you'd backtrack over completed tokens >> rather than characters. >> >> /Joe >> >> On Fri, Nov 13, 2015 at 3:02 PM, Sean Cribbs >> wrote: >> > Max, >> > >> > Do you have a link to your grammar? I can probably poke at it and give >> you >> > some tips. >> > >> > However, I am well aware of performance problems with neotoma -- with >> large >> > grammars or large inputs it drags. Yes, there are general problems for >> PEGs >> > in Erlang, but its current implementation is particularly naive and >> > wasteful. I'm working on a rewrite, but it's a complete overhaul (and >> more >> > faithful to the thesis and reference implementation "Pappy"). Since >> it's not >> > core to my day-job, I've only been able to work on the rewrite >> occasionally >> > in my free time. >> > >> > On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin >> wrote: >> >> >> >> Yes, Louis, I also think that there may be a simple way of speeding it >> up. >> >> >> >> I'm only afraid that I will have to open my university book and >> remember >> >> what LL-1 means and how it differs from LALR =) >> >> >> >> Ok, will try to profile it first. >> >> >> >> _______________________________________________ >> >> erlang-questions mailing list >> >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sat Nov 14 18:33:45 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 14 Nov 2015 20:33:45 +0300 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: <55A810C00B7CDA48A5AB43C566064B41014C52EAD0@SE-EX019.groupinfra.com> Message-ID: Got it. So, it is just a bad way of usage. I decided to put semantics and most of logic inside config to make validation as early as possible. And it seems that it is a misusage. OK, I will think what to do including option "ignore and relax" because only one user suffers from this problem. Thanks for hints and explanation! On Nov 14, 2015 8:20 PM, "Sean Cribbs" wrote: > Hi Max, > > My general impression of your grammar is that it conflates syntax and > semantics. Your configuration lines seem to be structured roughly like > KEYWORD VALUES* ( "{" BLOCK "}" )? ";", but you create a single rule for > every possible configuration item. This means at the top-level you have to > create ordered choices between completely unrelated extents. Every config > item you parse is going to go through the same backtracking among ~30 > alternatives, with nothing able to be memoized (because the stem of each > one is unique). This is a quintessential worst-case for the types of > parsers that neotoma currently generates. > > I suggest you rethink the parser, focus on what are the shared structures > and try to extract those. Use fewer rules, try not to treat each config > item as a special case, and then validate the structure after parsing it. > Alternatively, you could pick up and use cuttlefish which has a much > simpler configuration language and built-in semantic validation and > transformation and command-line tools. > > As you say earlier in the thread, you could also use leex and yecc. I > understand the desire not to, but their performance characteristics are > better known and more predictable. They might be worth it. > > I feel pretty responsible for the limitations you encountered. Ford's > thesis clearly outlines optimizations that neotoma does not do, including > more efficient matching of terminals via "switch", cost calculation to > determine what to memoize and inline, unrolling recognition logic instead > of using parser-combinators, and more. I'm working on a rewrite, but it's > not ready yet. > > I'm sorry. I hope the suggestions above help you focus the grammar into > something more usable. > > Sean > > On Sat, Nov 14, 2015 at 10:27 AM, Max Lapshin > wrote: > >> I've sent example from production to Sean. >> >> (flussonic@REDACTED)2> timer:tc(fun() -> >> config2_format:parse(element(2,file:read_file("big.conf"))), ok end). >> >> {29727926,ok} >> >> On Sat, Nov 14, 2015 at 4:05 PM, French, Michael >> wrote: >> >>> The concept of 'token' is fluid in PEGs. The terminal/non-terminal >>> distinction might not work. For example, the definition of 'alphanumeric' >>> might appear in several different 'tokens', rather than repeating char >>> classes, which means the token rules become non-terminals. >>> >>> Maybe use a hint in the grammar, like using an upper-case name for rule >>> LHS to indicate a 'token' (which is similar to a convention in Antlr). Then >>> always memo-ize the token rules, but not (necessarily) the others that have >>> lower-case rule names. >>> >>> BR >>> Mike >>> >>> ________________________________________ >>> From: erlang-questions-bounces@REDACTED [ >>> erlang-questions-bounces@REDACTED] on behalf of Joe Armstrong [ >>> erlang@REDACTED] >>> Sent: Friday, November 13, 2015 11:25 PM >>> To: Sean Cribbs >>> Cc: Erlang-Questions Questions >>> Subject: Re: [erlang-questions] speed of neotoma parser >>> >>> PEG parsers are notoriously inefficient. How about having a separate >>> tokenization pass, and parse token instead of characters. At a guess >>> this would be far faster since you'd backtrack over completed tokens >>> rather than characters. >>> >>> /Joe >>> >>> On Fri, Nov 13, 2015 at 3:02 PM, Sean Cribbs >>> wrote: >>> > Max, >>> > >>> > Do you have a link to your grammar? I can probably poke at it and give >>> you >>> > some tips. >>> > >>> > However, I am well aware of performance problems with neotoma -- with >>> large >>> > grammars or large inputs it drags. Yes, there are general problems for >>> PEGs >>> > in Erlang, but its current implementation is particularly naive and >>> > wasteful. I'm working on a rewrite, but it's a complete overhaul (and >>> more >>> > faithful to the thesis and reference implementation "Pappy"). Since >>> it's not >>> > core to my day-job, I've only been able to work on the rewrite >>> occasionally >>> > in my free time. >>> > >>> > On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin >>> wrote: >>> >> >>> >> Yes, Louis, I also think that there may be a simple way of speeding >>> it up. >>> >> >>> >> I'm only afraid that I will have to open my university book and >>> remember >>> >> what LL-1 means and how it differs from LALR =) >>> >> >>> >> Ok, will try to profile it first. >>> >> >>> >> _______________________________________________ >>> >> erlang-questions mailing list >>> >> erlang-questions@REDACTED >>> >> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >>> > >>> > >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> > >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sat Nov 14 19:44:19 2015 From: rvirding@REDACTED (Robert Virding) Date: Sat, 14 Nov 2015 19:44:19 +0100 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: <55A810C00B7CDA48A5AB43C566064B41014C52EAD0@SE-EX019.groupinfra.com> Message-ID: If you specifically want an LL(1) parser generator I have written one for Erlang and LFE, spell1, https://github.com/rvirding/spell1. You need a tokeniser for it, for example one generated with leex. The LALR(1) grammars of yecc are more general than LL(1) but spell1 does have a few benefits over yecc: - It is reentrant in that you don't need to pass in all the tokens in go but can keep adding them until it has gotten enough. - It can handle passing in too many tokens and just returns the left-overs. Yecc crashes on both of these cases. These aren't really a problem for erlang as you have the '.' which marks the end but I needed it for LFE. Robert On 14 November 2015 at 18:33, Max Lapshin wrote: > Got it. > > So, it is just a bad way of usage. > > I decided to put semantics and most of logic inside config to make > validation as early as possible. And it seems that it is a misusage. > > OK, I will think what to do including option "ignore and relax" because > only one user suffers from this problem. > > Thanks for hints and explanation! > On Nov 14, 2015 8:20 PM, "Sean Cribbs" wrote: > >> Hi Max, >> >> My general impression of your grammar is that it conflates syntax and >> semantics. Your configuration lines seem to be structured roughly like >> KEYWORD VALUES* ( "{" BLOCK "}" )? ";", but you create a single rule for >> every possible configuration item. This means at the top-level you have to >> create ordered choices between completely unrelated extents. Every config >> item you parse is going to go through the same backtracking among ~30 >> alternatives, with nothing able to be memoized (because the stem of each >> one is unique). This is a quintessential worst-case for the types of >> parsers that neotoma currently generates. >> >> I suggest you rethink the parser, focus on what are the shared structures >> and try to extract those. Use fewer rules, try not to treat each config >> item as a special case, and then validate the structure after parsing it. >> Alternatively, you could pick up and use cuttlefish which has a much >> simpler configuration language and built-in semantic validation and >> transformation and command-line tools. >> >> As you say earlier in the thread, you could also use leex and yecc. I >> understand the desire not to, but their performance characteristics are >> better known and more predictable. They might be worth it. >> >> I feel pretty responsible for the limitations you encountered. Ford's >> thesis clearly outlines optimizations that neotoma does not do, including >> more efficient matching of terminals via "switch", cost calculation to >> determine what to memoize and inline, unrolling recognition logic instead >> of using parser-combinators, and more. I'm working on a rewrite, but it's >> not ready yet. >> >> I'm sorry. I hope the suggestions above help you focus the grammar into >> something more usable. >> >> Sean >> >> On Sat, Nov 14, 2015 at 10:27 AM, Max Lapshin >> wrote: >> >>> I've sent example from production to Sean. >>> >>> (flussonic@REDACTED)2> timer:tc(fun() -> >>> config2_format:parse(element(2,file:read_file("big.conf"))), ok end). >>> >>> {29727926,ok} >>> >>> On Sat, Nov 14, 2015 at 4:05 PM, French, Michael >> > wrote: >>> >>>> The concept of 'token' is fluid in PEGs. The terminal/non-terminal >>>> distinction might not work. For example, the definition of 'alphanumeric' >>>> might appear in several different 'tokens', rather than repeating char >>>> classes, which means the token rules become non-terminals. >>>> >>>> Maybe use a hint in the grammar, like using an upper-case name for rule >>>> LHS to indicate a 'token' (which is similar to a convention in Antlr). Then >>>> always memo-ize the token rules, but not (necessarily) the others that have >>>> lower-case rule names. >>>> >>>> BR >>>> Mike >>>> >>>> ________________________________________ >>>> From: erlang-questions-bounces@REDACTED [ >>>> erlang-questions-bounces@REDACTED] on behalf of Joe Armstrong [ >>>> erlang@REDACTED] >>>> Sent: Friday, November 13, 2015 11:25 PM >>>> To: Sean Cribbs >>>> Cc: Erlang-Questions Questions >>>> Subject: Re: [erlang-questions] speed of neotoma parser >>>> >>>> PEG parsers are notoriously inefficient. How about having a separate >>>> tokenization pass, and parse token instead of characters. At a guess >>>> this would be far faster since you'd backtrack over completed tokens >>>> rather than characters. >>>> >>>> /Joe >>>> >>>> On Fri, Nov 13, 2015 at 3:02 PM, Sean Cribbs >>>> wrote: >>>> > Max, >>>> > >>>> > Do you have a link to your grammar? I can probably poke at it and >>>> give you >>>> > some tips. >>>> > >>>> > However, I am well aware of performance problems with neotoma -- with >>>> large >>>> > grammars or large inputs it drags. Yes, there are general problems >>>> for PEGs >>>> > in Erlang, but its current implementation is particularly naive and >>>> > wasteful. I'm working on a rewrite, but it's a complete overhaul (and >>>> more >>>> > faithful to the thesis and reference implementation "Pappy"). Since >>>> it's not >>>> > core to my day-job, I've only been able to work on the rewrite >>>> occasionally >>>> > in my free time. >>>> > >>>> > On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin >>>> wrote: >>>> >> >>>> >> Yes, Louis, I also think that there may be a simple way of speeding >>>> it up. >>>> >> >>>> >> I'm only afraid that I will have to open my university book and >>>> remember >>>> >> what LL-1 means and how it differs from LALR =) >>>> >> >>>> >> Ok, will try to profile it first. >>>> >> >>>> >> _______________________________________________ >>>> >> erlang-questions mailing list >>>> >> erlang-questions@REDACTED >>>> >> http://erlang.org/mailman/listinfo/erlang-questions >>>> >> >>>> > >>>> > >>>> > _______________________________________________ >>>> > erlang-questions mailing list >>>> > erlang-questions@REDACTED >>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>> > >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sat Nov 14 19:47:27 2015 From: rvirding@REDACTED (Robert Virding) Date: Sat, 14 Nov 2015 19:47:27 +0100 Subject: [erlang-questions] Languages running on BEAM In-Reply-To: <5644BAEC.9090609@llaisdy.com> References: <5640B775.1090700@llaisdy.com> <5644BAEC.9090609@llaisdy.com> Message-ID: I think an objective comparison would be very difficult as some of the differences are of a religious nature, for examle lisp-1 vs lisp-2 :-), and some are just a matter of preference. You basically go with the one that feels the best. Robert On 12 November 2015 at 17:14, Ivan Uemlianin wrote: > Dear Robert > > Thanks for your comment. > > I think an objective comparison (i.e. by a fourth party) of the three > different erlang lisps will definitely be in order. > > Best wishes > > Ivan > > > > On 12/11/2015 16:06, Robert Virding wrote: > > I would point out that Eric Merrit was very wrong about aspects of LFE in > his comparison with Joxa which makes some parts of it not very meaningful. > > Robert > > > On 9 November 2015 at 16:10, Ivan Uemlianin wrote: > >> Dear All >> >> I am compiling a list of programming languages that run on BEAM, the >> Erlang VM: >> >> https://github.com/llaisdy/beam_languages >> >> The resource is aimed at people who might be thinking of implementing >> such a language themselves. Consequently the list includes experimental >> and abandoned languages as well as the stars. >> >> I am also including a section listing documents (papers, presentations, >> blog posts, etc.) that cover the theme of implementing languages for the >> BEAM in general. >> >> Languages listed so far: >> >> - efene >> - elixir >> - erl2 >> - erlang >> - erlog >> - erlyjs >> - haskerl >> - joxa >> - lfe >> - luerl >> - reia >> - 'the concurrent schemer' >> - xerl >> >> If you know about a language or a document that should be on one of these >> lists --- or if I've made mistakes! ---, please let me know (email, >> issue/PR, tweet, etc.). >> >> With thanks and best wishes >> >> Ivan >> >> >> -- >> ============================================================ >> Ivan A. Uemlianin PhD >> Llaisdy >> Speech Technology Research and Development >> >> ivan@REDACTED >> @llaisdy >> llaisdy.wordpress.com >> github.com/llaisdy >> www.linkedin.com/in/ivanuemlianin >> >> festina lente >> ============================================================ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > @llaisdy > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > festina lente > ============================================================ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Sat Nov 14 20:37:38 2015 From: ivan@REDACTED (Ivan Uemlianin) Date: Sat, 14 Nov 2015 19:37:38 +0000 Subject: [erlang-questions] Languages running on BEAM In-Reply-To: References: <5640B775.1090700@llaisdy.com> <5644BAEC.9090609@llaisdy.com> Message-ID: <1447529860664-40035f66-33490bd8-5470d99d@llaisdy.com> Sorry, I didn't mean comparison as in better/worse, I just meant laying out the differences (Also, if ever such a comparison were done, it probably wouldn't be by me;). My objevtive in compiling the list is to help me think through language implementation. Ivan On Sat, Nov 14, 2015 at 6:47 pm, Robert Virding wrote: I think an objective comparison would be very difficult as some of the differences are of a religious nature, for examle lisp-1 vs lisp-2 :-), and some are just a matter of preference. You basically go with the one that feels the best. Robert On 12 November 2015 at 17:14, Ivan Uemlianin < ivan@REDACTED [ivan@REDACTED] > wrote: Dear Robert Thanks for your comment. I think an objective comparison (i.e. by a fourth party) of the three different erlang lisps will definitely be in order. Best wishes Ivan On 12/11/2015 16:06, Robert Virding wrote: I would point out that Eric Merrit was very wrong about aspects of LFE in his comparison with Joxa which makes some parts of it not very meaningful. Robert On 9 November 2015 at 16:10, Ivan Uemlianin < ivan@REDACTED [ivan@REDACTED] > wrote: Dear All I am compiling a list of programming languages that run on BEAM, the Erlang VM: [https://github.com/llaisdy/beam_languages] https://github.com/llaisdy/beam_languages [https://github.com/llaisdy/beam_languages] The resource is aimed at people who might be thinking of implementing such a language themselves. Consequently the list includes experimental and abandoned languages as well as the stars. I am also including a section listing documents (papers, presentations, blog posts, etc.) that cover the theme of implementing languages for the BEAM in general. Languages listed so far: - efene - elixir - erl2 - erlang - erlog - erlyjs - haskerl - joxa - lfe - luerl - reia - 'the concurrent schemer' - xerl If you know about a language or a document that should be on one of these lists --- or if I've made mistakes! ---, please let me know (email, issue/PR, tweet, etc.). With thanks and best wishes Ivan -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED [ivan@REDACTED] @llaisdy llaisdy.wordpress.com [http://llaisdy.wordpress.com] github.com/llaisdy [http://github.com/llaisdy] [http://www.linkedin.com/in/ivanuemlianin] www.linkedin.com/in/ivanuemlianin [http://www.linkedin.com/in/ivanuemlianin] festina lente ============================================================ _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED [erlang-questions@REDACTED] [http://erlang.org/mailman/listinfo/erlang-questions] http://erlang.org/mailman/listinfo/erlang-questions [http://erlang.org/mailman/listinfo/erlang-questions] -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED [ivan@REDACTED] @llaisdy llaisdy.wordpress.com [http://llaisdy.wordpress.com]github.com/llaisdy [http://github.com/llaisdy] [http://www.linkedin.com/in/ivanuemlianin]www.linkedin.com/in/ivanuemlianin [http://www.linkedin.com/in/ivanuemlianin] festina lente ============================================================ -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Sat Nov 14 21:28:02 2015 From: serge@REDACTED (Serge Aleynikov) Date: Sat, 14 Nov 2015 15:28:02 -0500 Subject: [erlang-questions] erlang:system_time() bug Message-ID: We noticed that erlang:system_time() returns unexpected results on Windows (v10 and v7 were tested, while working correctly on Linux: Windows: 1> erlang:system_time(). 1482272700485632 2> erlang:now(). {1447,531971,381000} Linux: 1> erlang:now(). {1447,531817,353669} 2> erlang:system_time(). 1447531823437962212 Note the megaseconds part of the system_time/0 call on Windows is 148... whereas it should be 144... While the bug report was added to bugs.erlang.org, I am wondering if this is a known issue or there's some tweaking related to new time management functions that needs to be done on Windows? Serge -------------- next part -------------- An HTML attachment was scrubbed... URL: From semmitmondo@REDACTED Sat Nov 14 22:39:36 2015 From: semmitmondo@REDACTED (semmit mondo) Date: Sat, 14 Nov 2015 22:39:36 +0100 (CET) Subject: [erlang-questions] hwo to execute a text editor Message-ID: Hi, I work on a command line tool (powered by escript) that needs to launch atext editor (vim for instance) at a certain point. I tried to do something like os:cmd('vim xxx'). It works almost fine, except that vim isn't connected to the terminal whileit's running. Insted, vim is connected to the erlang process, and I cannotsee anything in the terminal window and cannot quit vim with ":q", or anyother keys, because keypresses are not sent to the proper process. How can I start a process and allow it to have control over the terminaland ask the VM to wait until the subprocess terminates? S. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sat Nov 14 23:00:25 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 15 Nov 2015 01:00:25 +0300 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: <55A810C00B7CDA48A5AB43C566064B41014C52EAD0@SE-EX019.groupinfra.com> Message-ID: I need parsing of a nginx-like config. In my current implementation I have described almost all keywords in peg grammar, so any unknown keyword will stop parsing. If I have correctly understood Sean, better will be to validate after parsing. I need to step back and read about lalr vs ll difference first. On Nov 14, 2015 9:44 PM, "Robert Virding" wrote: > If you specifically want an LL(1) parser generator I have written one for > Erlang and LFE, spell1, https://github.com/rvirding/spell1. You need a > tokeniser for it, for example one generated with leex. The LALR(1) grammars > of yecc are more general than LL(1) but spell1 does have a few benefits > over yecc: > > - It is reentrant in that you don't need to pass in all the tokens in go > but can keep adding them until it has gotten enough. > - It can handle passing in too many tokens and just returns the left-overs. > > Yecc crashes on both of these cases. These aren't really a problem for > erlang as you have the '.' which marks the end but I needed it for LFE. > > Robert > > > On 14 November 2015 at 18:33, Max Lapshin wrote: > >> Got it. >> >> So, it is just a bad way of usage. >> >> I decided to put semantics and most of logic inside config to make >> validation as early as possible. And it seems that it is a misusage. >> >> OK, I will think what to do including option "ignore and relax" because >> only one user suffers from this problem. >> >> Thanks for hints and explanation! >> On Nov 14, 2015 8:20 PM, "Sean Cribbs" wrote: >> >>> Hi Max, >>> >>> My general impression of your grammar is that it conflates syntax and >>> semantics. Your configuration lines seem to be structured roughly like >>> KEYWORD VALUES* ( "{" BLOCK "}" )? ";", but you create a single rule for >>> every possible configuration item. This means at the top-level you have to >>> create ordered choices between completely unrelated extents. Every config >>> item you parse is going to go through the same backtracking among ~30 >>> alternatives, with nothing able to be memoized (because the stem of each >>> one is unique). This is a quintessential worst-case for the types of >>> parsers that neotoma currently generates. >>> >>> I suggest you rethink the parser, focus on what are the shared >>> structures and try to extract those. Use fewer rules, try not to treat each >>> config item as a special case, and then validate the structure after >>> parsing it. Alternatively, you could pick up and use cuttlefish which has a >>> much simpler configuration language and built-in semantic validation and >>> transformation and command-line tools. >>> >>> As you say earlier in the thread, you could also use leex and yecc. I >>> understand the desire not to, but their performance characteristics are >>> better known and more predictable. They might be worth it. >>> >>> I feel pretty responsible for the limitations you encountered. Ford's >>> thesis clearly outlines optimizations that neotoma does not do, including >>> more efficient matching of terminals via "switch", cost calculation to >>> determine what to memoize and inline, unrolling recognition logic instead >>> of using parser-combinators, and more. I'm working on a rewrite, but it's >>> not ready yet. >>> >>> I'm sorry. I hope the suggestions above help you focus the grammar into >>> something more usable. >>> >>> Sean >>> >>> On Sat, Nov 14, 2015 at 10:27 AM, Max Lapshin >>> wrote: >>> >>>> I've sent example from production to Sean. >>>> >>>> (flussonic@REDACTED)2> timer:tc(fun() -> >>>> config2_format:parse(element(2,file:read_file("big.conf"))), ok end). >>>> >>>> {29727926,ok} >>>> >>>> On Sat, Nov 14, 2015 at 4:05 PM, French, Michael < >>>> michael.french@REDACTED> wrote: >>>> >>>>> The concept of 'token' is fluid in PEGs. The terminal/non-terminal >>>>> distinction might not work. For example, the definition of 'alphanumeric' >>>>> might appear in several different 'tokens', rather than repeating char >>>>> classes, which means the token rules become non-terminals. >>>>> >>>>> Maybe use a hint in the grammar, like using an upper-case name for >>>>> rule LHS to indicate a 'token' (which is similar to a convention in Antlr). >>>>> Then always memo-ize the token rules, but not (necessarily) the others that >>>>> have lower-case rule names. >>>>> >>>>> BR >>>>> Mike >>>>> >>>>> ________________________________________ >>>>> From: erlang-questions-bounces@REDACTED [ >>>>> erlang-questions-bounces@REDACTED] on behalf of Joe Armstrong [ >>>>> erlang@REDACTED] >>>>> Sent: Friday, November 13, 2015 11:25 PM >>>>> To: Sean Cribbs >>>>> Cc: Erlang-Questions Questions >>>>> Subject: Re: [erlang-questions] speed of neotoma parser >>>>> >>>>> PEG parsers are notoriously inefficient. How about having a separate >>>>> tokenization pass, and parse token instead of characters. At a guess >>>>> this would be far faster since you'd backtrack over completed tokens >>>>> rather than characters. >>>>> >>>>> /Joe >>>>> >>>>> On Fri, Nov 13, 2015 at 3:02 PM, Sean Cribbs >>>>> wrote: >>>>> > Max, >>>>> > >>>>> > Do you have a link to your grammar? I can probably poke at it and >>>>> give you >>>>> > some tips. >>>>> > >>>>> > However, I am well aware of performance problems with neotoma -- >>>>> with large >>>>> > grammars or large inputs it drags. Yes, there are general problems >>>>> for PEGs >>>>> > in Erlang, but its current implementation is particularly naive and >>>>> > wasteful. I'm working on a rewrite, but it's a complete overhaul >>>>> (and more >>>>> > faithful to the thesis and reference implementation "Pappy"). Since >>>>> it's not >>>>> > core to my day-job, I've only been able to work on the rewrite >>>>> occasionally >>>>> > in my free time. >>>>> > >>>>> > On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin >>>>> wrote: >>>>> >> >>>>> >> Yes, Louis, I also think that there may be a simple way of speeding >>>>> it up. >>>>> >> >>>>> >> I'm only afraid that I will have to open my university book and >>>>> remember >>>>> >> what LL-1 means and how it differs from LALR =) >>>>> >> >>>>> >> Ok, will try to profile it first. >>>>> >> >>>>> >> _______________________________________________ >>>>> >> erlang-questions mailing list >>>>> >> erlang-questions@REDACTED >>>>> >> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >> >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > erlang-questions mailing list >>>>> > erlang-questions@REDACTED >>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>> > >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>>> >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Sun Nov 15 06:08:33 2015 From: serge@REDACTED (Serge Aleynikov) Date: Sun, 15 Nov 2015 00:08:33 -0500 Subject: [erlang-questions] speed of neotoma parser In-Reply-To: References: <55A810C00B7CDA48A5AB43C566064B41014C52EAD0@SE-EX019.groupinfra.com> Message-ID: Max, You can try this parser, which does parse config syntax very similar to what you need: https://github.com/saleyn/erlcfg Regards, Serge On Sat, Nov 14, 2015 at 5:00 PM, Max Lapshin wrote: > I need parsing of a nginx-like config. > In my current implementation I have described almost all keywords in peg > grammar, so any unknown keyword will stop parsing. > > If I have correctly understood Sean, better will be to validate after > parsing. > > I need to step back and read about lalr vs ll difference first. > On Nov 14, 2015 9:44 PM, "Robert Virding" wrote: > >> If you specifically want an LL(1) parser generator I have written one for >> Erlang and LFE, spell1, https://github.com/rvirding/spell1. You need a >> tokeniser for it, for example one generated with leex. The LALR(1) grammars >> of yecc are more general than LL(1) but spell1 does have a few benefits >> over yecc: >> >> - It is reentrant in that you don't need to pass in all the tokens in go >> but can keep adding them until it has gotten enough. >> - It can handle passing in too many tokens and just returns the >> left-overs. >> >> Yecc crashes on both of these cases. These aren't really a problem for >> erlang as you have the '.' which marks the end but I needed it for LFE. >> >> Robert >> >> >> On 14 November 2015 at 18:33, Max Lapshin wrote: >> >>> Got it. >>> >>> So, it is just a bad way of usage. >>> >>> I decided to put semantics and most of logic inside config to make >>> validation as early as possible. And it seems that it is a misusage. >>> >>> OK, I will think what to do including option "ignore and relax" because >>> only one user suffers from this problem. >>> >>> Thanks for hints and explanation! >>> On Nov 14, 2015 8:20 PM, "Sean Cribbs" wrote: >>> >>>> Hi Max, >>>> >>>> My general impression of your grammar is that it conflates syntax and >>>> semantics. Your configuration lines seem to be structured roughly like >>>> KEYWORD VALUES* ( "{" BLOCK "}" )? ";", but you create a single rule for >>>> every possible configuration item. This means at the top-level you have to >>>> create ordered choices between completely unrelated extents. Every config >>>> item you parse is going to go through the same backtracking among ~30 >>>> alternatives, with nothing able to be memoized (because the stem of each >>>> one is unique). This is a quintessential worst-case for the types of >>>> parsers that neotoma currently generates. >>>> >>>> I suggest you rethink the parser, focus on what are the shared >>>> structures and try to extract those. Use fewer rules, try not to treat each >>>> config item as a special case, and then validate the structure after >>>> parsing it. Alternatively, you could pick up and use cuttlefish which has a >>>> much simpler configuration language and built-in semantic validation and >>>> transformation and command-line tools. >>>> >>>> As you say earlier in the thread, you could also use leex and yecc. I >>>> understand the desire not to, but their performance characteristics are >>>> better known and more predictable. They might be worth it. >>>> >>>> I feel pretty responsible for the limitations you encountered. Ford's >>>> thesis clearly outlines optimizations that neotoma does not do, including >>>> more efficient matching of terminals via "switch", cost calculation to >>>> determine what to memoize and inline, unrolling recognition logic instead >>>> of using parser-combinators, and more. I'm working on a rewrite, but it's >>>> not ready yet. >>>> >>>> I'm sorry. I hope the suggestions above help you focus the grammar into >>>> something more usable. >>>> >>>> Sean >>>> >>>> On Sat, Nov 14, 2015 at 10:27 AM, Max Lapshin >>>> wrote: >>>> >>>>> I've sent example from production to Sean. >>>>> >>>>> (flussonic@REDACTED)2> timer:tc(fun() -> >>>>> config2_format:parse(element(2,file:read_file("big.conf"))), ok end). >>>>> >>>>> {29727926,ok} >>>>> >>>>> On Sat, Nov 14, 2015 at 4:05 PM, French, Michael < >>>>> michael.french@REDACTED> wrote: >>>>> >>>>>> The concept of 'token' is fluid in PEGs. The terminal/non-terminal >>>>>> distinction might not work. For example, the definition of 'alphanumeric' >>>>>> might appear in several different 'tokens', rather than repeating char >>>>>> classes, which means the token rules become non-terminals. >>>>>> >>>>>> Maybe use a hint in the grammar, like using an upper-case name for >>>>>> rule LHS to indicate a 'token' (which is similar to a convention in Antlr). >>>>>> Then always memo-ize the token rules, but not (necessarily) the others that >>>>>> have lower-case rule names. >>>>>> >>>>>> BR >>>>>> Mike >>>>>> >>>>>> ________________________________________ >>>>>> From: erlang-questions-bounces@REDACTED [ >>>>>> erlang-questions-bounces@REDACTED] on behalf of Joe Armstrong [ >>>>>> erlang@REDACTED] >>>>>> Sent: Friday, November 13, 2015 11:25 PM >>>>>> To: Sean Cribbs >>>>>> Cc: Erlang-Questions Questions >>>>>> Subject: Re: [erlang-questions] speed of neotoma parser >>>>>> >>>>>> PEG parsers are notoriously inefficient. How about having a separate >>>>>> tokenization pass, and parse token instead of characters. At a guess >>>>>> this would be far faster since you'd backtrack over completed tokens >>>>>> rather than characters. >>>>>> >>>>>> /Joe >>>>>> >>>>>> On Fri, Nov 13, 2015 at 3:02 PM, Sean Cribbs >>>>>> wrote: >>>>>> > Max, >>>>>> > >>>>>> > Do you have a link to your grammar? I can probably poke at it and >>>>>> give you >>>>>> > some tips. >>>>>> > >>>>>> > However, I am well aware of performance problems with neotoma -- >>>>>> with large >>>>>> > grammars or large inputs it drags. Yes, there are general problems >>>>>> for PEGs >>>>>> > in Erlang, but its current implementation is particularly naive and >>>>>> > wasteful. I'm working on a rewrite, but it's a complete overhaul >>>>>> (and more >>>>>> > faithful to the thesis and reference implementation "Pappy"). Since >>>>>> it's not >>>>>> > core to my day-job, I've only been able to work on the rewrite >>>>>> occasionally >>>>>> > in my free time. >>>>>> > >>>>>> > On Thu, Nov 12, 2015 at 12:07 PM, Max Lapshin < >>>>>> max.lapshin@REDACTED> wrote: >>>>>> >> >>>>>> >> Yes, Louis, I also think that there may be a simple way of >>>>>> speeding it up. >>>>>> >> >>>>>> >> I'm only afraid that I will have to open my university book and >>>>>> remember >>>>>> >> what LL-1 means and how it differs from LALR =) >>>>>> >> >>>>>> >> Ok, will try to profile it first. >>>>>> >> >>>>>> >> _______________________________________________ >>>>>> >> erlang-questions mailing list >>>>>> >> erlang-questions@REDACTED >>>>>> >> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >> >>>>>> > >>>>>> > >>>>>> > _______________________________________________ >>>>>> > erlang-questions mailing list >>>>>> > erlang-questions@REDACTED >>>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>>> > >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>> >>>>> >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jargon@REDACTED Sun Nov 15 10:01:49 2015 From: jargon@REDACTED (Johannes =?utf-8?B?V2Vpw59s?=) Date: Sun, 15 Nov 2015 10:01:49 +0100 Subject: [erlang-questions] Patch Package OTP 18.1.4 Released In-Reply-To: <5645BAFE.2010604@ericsson.com> References: <5645BAFE.2010604@ericsson.com> Message-ID: <20151115090149.GA5907@molb.org> On Fri, Nov 13, 2015 at 11:27AM +0100, Zandra Hird wrote: > ? OTP-13061??? Application(s): inets > > ?????????????? Improved error handling and gracfully termination when > ?????????????? an invalid chunked length header is encountered. This introduced an incompatible change which breaks a part of our application: Prior to this the HTTP client could parse server responses that have whitespace after the chunk size (some embedded legacy devices still do this), after OTP-13061 the HTTP handler crashes with "bad_return_value". I think this did not happen on purpose, it was just a side effect of replacing the custom hexlist_to_integer/1 with list_to_integer/2. I opened two pull requests that fix the problem for us: - https://github.com/erlang/otp/pull/894 - https://github.com/erlang/otp/pull/895 Regards, Johannes From ok@REDACTED Mon Nov 16 01:09:19 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 16 Nov 2015 13:09:19 +1300 Subject: [erlang-questions] Languages running on BEAM In-Reply-To: References: <5640B775.1090700@llaisdy.com> <5644BAEC.9090609@llaisdy.com> Message-ID: <50B15BF1-6134-4F46-9142-87A96E8CBB73@cs.otago.ac.nz> On 15/11/2015, at 7:47 am, Robert Virding wrote: > I think an objective comparison would be very difficult as some of the differences are of a religious nature, for examle lisp-1 vs lisp-2 :-), and some are just a matter of preference. There are two different things: comparison and evaluation. You say "X has U but not V, Y has V but not U" without revealing which you think is better. Mind you, the last language comparison I looked at drew its data from RosettaCode.org. As a learning exercise, I've been implementing as many RosettaCode tasks as I can in a language which need not concern us here. The point of interest is that while I still think this is useful as a training exercise, I have ceased to believe that it tells us much about languages. One *major* source of differences is not languages but *libraries*. For example, "Play a sound" is trivial if you are allowed to do system("audioplay foobar.wav") which hardly tells you much about the language. Since Elixir, LFE, and Erlang can all call the same libraries, I don't really expect much difference between them at the *system* level. I do expect a difference in coding errors, which will depend on the verbosity of the language, the 'trickiness' of the language, the previous experience of the programmer, and tool support, not to mention how much you depend on code generation vs hand coding. Again, in my own RosettaCode exercises, I have stumbled over *library* issues (like the predefined all-subsets-of-a-set function returning a stretchy array when I expected a _set_ of sets) more often than *language* issues. Something I've always believed, but the RosettaCode study has really driven home to me, is how much variation there is between programmers and even *within* programmers: should I go for the one-liner that makes strong assumptions about its input or do something longer but safer? In a contest I might take short cuts, but in real programming I ought to play safe. From roger@REDACTED Mon Nov 16 11:21:35 2015 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 16 Nov 2015 10:21:35 +0000 Subject: [erlang-questions] hwo to execute a text editor In-Reply-To: References: Message-ID: On 14 November 2015 at 21:39, semmit mondo wrote: > How can I start a process and allow it to have control over the terminal > and ask the VM to wait until the subprocess terminates? https://github.com/saleyn/erlexec (docs at http://saleyn.github.io/erlexec/) has "Running interactive processes with psudo-terminal pty support." Disclaimer: I've used erlexec for other things, but not for the pty support. From bjorn@REDACTED Mon Nov 16 13:37:46 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 16 Nov 2015 13:37:46 +0100 Subject: [erlang-questions] EEP 45: New macros for function name and arity Message-ID: This is the fourth draft of EEP 45. http://www.erlang.org/eeps/eep-0045.html https://github.com/erlang/eep/blob/master/eeps/eep-0045.md -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From rickard@REDACTED Mon Nov 16 16:35:29 2015 From: rickard@REDACTED (Rickard Green) Date: Mon, 16 Nov 2015 16:35:29 +0100 Subject: [erlang-questions] erlang:system_time() bug In-Reply-To: References: Message-ID: On Sat, Nov 14, 2015 at 9:28 PM, Serge Aleynikov wrote: > We noticed that erlang:system_time() returns unexpected results on Windows > (v10 and v7 were tested, while working correctly on Linux: > > Windows: > 1> erlang:system_time(). > 1482272700485632 > 2> erlang:now(). > {1447,531971,381000} > > Linux: > 1> erlang:now(). > {1447,531817,353669} > 2> erlang:system_time(). > 1447531823437962212 > > Note the megaseconds part of the system_time/0 call on Windows is 148... > whereas it should be 144... > > While the bug report was added to bugs.erlang.org, I am wondering if this is > a known issue or there's some tweaking related to new time management > functions that needs to be done on Windows? > > Serge > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > This is not a bug. For more info see . Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB From vittico@REDACTED Mon Nov 16 15:58:11 2015 From: vittico@REDACTED (Victor Medina) Date: Mon, 16 Nov 2015 10:28:11 -0430 Subject: [erlang-questions] DTLS 1.2 / Erlang Support Message-ID: Hi guys! Im starting to build a WebRTC app using Erlang. Im using a custom build of OTP 17.5, but using the distro's OpenSSL library (currently OpenSSL 1.0.1k 8 Jan 2015). As far as I know, DTLS 1.2 is not supported on OpenSSL 1.0.1k, its only suported on 1.0.2 onward. So, at least Ill need to update OpenSSL or buid OTP against a new OpenSSL version. Does OTP 17.5 supoorts DTLS 1.2 if the underlying OpenSSLsupports it? Any advice? Sin mas a que hacer referencia, Victor Medina -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean@REDACTED Mon Nov 16 18:00:01 2015 From: sean@REDACTED (Functional Jobs) Date: Mon, 16 Nov 2015 12:00:01 -0500 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <564a0bc3b2220@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Software Engineer - Functional Programmer - Erlang at Change Healthcare https://functionaljobs.com/jobs/8866-software-engineer-functional-programmer-erlang-at-change-healthcare Cheers, Sean Murphy FunctionalJobs.com From max.lapshin@REDACTED Mon Nov 16 18:06:33 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 16 Nov 2015 20:06:33 +0300 Subject: [erlang-questions] DTLS 1.2 / Erlang Support In-Reply-To: References: Message-ID: Sorry, but hahahaha You need to find for a dtls4srtp.zip, fix it a bit, launch and then try to launch srtp.erl Erlang doesn't take "ssl" part of openssl, only "crypto". Dtls must be implemented in erlang. If you wail to, then consider sending lot of flowers to Ingela, perhaps she would implement it a bit earlier =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Mon Nov 16 18:38:14 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 16 Nov 2015 17:38:14 +0000 Subject: [erlang-questions] DTLS 1.2 / Erlang Support In-Reply-To: References: Message-ID: other way would be a driver or port over openwebrtc :) On Mon, 16 Nov 2015 at 18:06, Max Lapshin wrote: > Sorry, but hahahaha > > You need to find for a dtls4srtp.zip, fix it a bit, launch and then try to > launch srtp.erl > > Erlang doesn't take "ssl" part of openssl, only "crypto". Dtls must be > implemented in erlang. > > If you wail to, then consider sending lot of flowers to Ingela, perhaps > she would implement it a bit earlier =) > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Mon Nov 16 18:50:48 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 16 Nov 2015 17:50:48 +0000 Subject: [erlang-questions] DTLS 1.2 / Erlang Support In-Reply-To: References: Message-ID: have a look to: http://www.openwebrtc.org/blog/2014/10/31/webrtc-in-safari-using-openwebrtc https://github.com/EricssonResearch/openwebrtc-browser-extensions they have a daemon apparently. On Mon, 16 Nov 2015 at 18:45, Victor Medina wrote: > > On Mon, Nov 16, 2015 at 1:08 PM, Benoit Chesneau > wrote: > >> openwebrtc > > > um... interesting! taking a look.... > > Other would be just to do something in c++ or c that takes care of > DTLS(using wolf's or mbed's tls) and handle it back to a erlang process... > > > Sin mas a que hacer referencia, > > Victor Medina > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Mon Nov 16 19:28:46 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 16 Nov 2015 18:28:46 +0000 Subject: [erlang-questions] DTLS 1.2 / Erlang Support In-Reply-To: References: Message-ID: cool :) keep us updated On Mon, 16 Nov 2015 at 18:54, Victor Medina wrote: > In fact Im clonning it! =) > > > Sin mas a que hacer referencia, > > Victor Medina > > On Mon, Nov 16, 2015 at 1:20 PM, Benoit Chesneau > wrote: > >> have a look to: >> >> http://www.openwebrtc.org/blog/2014/10/31/webrtc-in-safari-using-openwebrtc >> >> https://github.com/EricssonResearch/openwebrtc-browser-extensions >> >> they have a daemon apparently. >> On Mon, 16 Nov 2015 at 18:45, Victor Medina wrote: >> >>> >>> On Mon, Nov 16, 2015 at 1:08 PM, Benoit Chesneau >>> wrote: >>> >>>> openwebrtc >>> >>> >>> um... interesting! taking a look.... >>> >>> Other would be just to do something in c++ or c that takes care of >>> DTLS(using wolf's or mbed's tls) and handle it back to a erlang process... >>> >>> >>> Sin mas a que hacer referencia, >>> >>> Victor Medina >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vittico@REDACTED Mon Nov 16 18:09:34 2015 From: vittico@REDACTED (Victor Medina) Date: Mon, 16 Nov 2015 12:39:34 -0430 Subject: [erlang-questions] DTLS 1.2 / Erlang Support In-Reply-To: References: Message-ID: ?? flowers and chocolates if she wants El 16/11/2015 12:36, "Max Lapshin" escribi?: > > Sorry, but hahahaha > > You need to find for a dtls4srtp.zip, fix it a bit, launch and then try to launch srtp.erl > > Erlang doesn't take "ssl" part of openssl, only "crypto". Dtls must be implemented in erlang. > > If you wail to, then consider sending lot of flowers to Ingela, perhaps she would implement it a bit earlier =) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vittico@REDACTED Mon Nov 16 18:44:41 2015 From: vittico@REDACTED (Victor Medina) Date: Mon, 16 Nov 2015 13:14:41 -0430 Subject: [erlang-questions] DTLS 1.2 / Erlang Support In-Reply-To: References: Message-ID: On Mon, Nov 16, 2015 at 1:08 PM, Benoit Chesneau wrote: > openwebrtc um... interesting! taking a look.... Other would be just to do something in c++ or c that takes care of DTLS(using wolf's or mbed's tls) and handle it back to a erlang process... Sin mas a que hacer referencia, Victor Medina -------------- next part -------------- An HTML attachment was scrubbed... URL: From vittico@REDACTED Mon Nov 16 18:54:15 2015 From: vittico@REDACTED (Victor Medina) Date: Mon, 16 Nov 2015 13:24:15 -0430 Subject: [erlang-questions] DTLS 1.2 / Erlang Support In-Reply-To: References: Message-ID: In fact Im clonning it! =) Sin mas a que hacer referencia, Victor Medina On Mon, Nov 16, 2015 at 1:20 PM, Benoit Chesneau wrote: > have a look to: > http://www.openwebrtc.org/blog/2014/10/31/webrtc-in-safari-using-openwebrtc > > https://github.com/EricssonResearch/openwebrtc-browser-extensions > > they have a daemon apparently. > On Mon, 16 Nov 2015 at 18:45, Victor Medina wrote: > >> >> On Mon, Nov 16, 2015 at 1:08 PM, Benoit Chesneau >> wrote: >> >>> openwebrtc >> >> >> um... interesting! taking a look.... >> >> Other would be just to do something in c++ or c that takes care of >> DTLS(using wolf's or mbed's tls) and handle it back to a erlang process... >> >> >> Sin mas a que hacer referencia, >> >> Victor Medina >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vittico@REDACTED Mon Nov 16 19:34:10 2015 From: vittico@REDACTED (Victor Medina) Date: Mon, 16 Nov 2015 14:04:10 -0430 Subject: [erlang-questions] DTLS 1.2 / Erlang Support In-Reply-To: References: Message-ID: altough I still prefer sending chocolates and flowers =) Sin mas a que hacer referencia, Victor Medina On Mon, Nov 16, 2015 at 1:58 PM, Benoit Chesneau wrote: > cool :) keep us updated > > On Mon, 16 Nov 2015 at 18:54, Victor Medina wrote: > >> In fact Im clonning it! =) >> >> >> Sin mas a que hacer referencia, >> >> Victor Medina >> >> On Mon, Nov 16, 2015 at 1:20 PM, Benoit Chesneau >> wrote: >> >>> have a look to: >>> >>> http://www.openwebrtc.org/blog/2014/10/31/webrtc-in-safari-using-openwebrtc >>> >>> https://github.com/EricssonResearch/openwebrtc-browser-extensions >>> >>> they have a daemon apparently. >>> On Mon, 16 Nov 2015 at 18:45, Victor Medina wrote: >>> >>>> >>>> On Mon, Nov 16, 2015 at 1:08 PM, Benoit Chesneau >>>> wrote: >>>> >>>>> openwebrtc >>>> >>>> >>>> um... interesting! taking a look.... >>>> >>>> Other would be just to do something in c++ or c that takes care of >>>> DTLS(using wolf's or mbed's tls) and handle it back to a erlang process... >>>> >>>> >>>> Sin mas a que hacer referencia, >>>> >>>> Victor Medina >>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Tue Nov 17 10:17:50 2015 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 17 Nov 2015 01:17:50 -0800 Subject: [erlang-questions] max_heap_size Message-ID: <564AF0BE.9070403@gmail.com> Could a max_heap_size value be added for enforcing a limitation on the growth a process' heap, preventing it from crashing the Erlang VM? This isn't already implemented, right? Thanks, Michael From roshhnikita@REDACTED Tue Nov 17 11:55:21 2015 From: roshhnikita@REDACTED (=?UTF-8?B?0J3QuNC60LjRgtCwINCg0L7RidGD0L/QutC40L0=?=) Date: Tue, 17 Nov 2015 16:55:21 +0600 Subject: [erlang-questions] Flag not documented but give speed up. Message-ID: Why don't we use the flag which is not documented and not used by default when "+native" flag is enabled? This is table comparison https://drive.google.com/open?id=1P3nYD0EZXL16TqQ0WStqa3RqWt6riqd9MH25YsmnGmE There is +strict_record_updates flag in Erlang which is used to disable setelement for record updating. In this case get_tuple_element, put and put_tuple are used instead of setelement. Unlike setelement, they create new record with applied changes. For example, I wrote a simple method (function) working with record, and translated it to a native code. By the way, as we can see in this table, the +strict_record_updates flag achieves greater speedup when used with +native flag. Likewise, the "put" flag is better than "setelement" when used with +native. -module(testing). -compile(export_all). -define(COUNT, 100000000). -record(big_record, { id1 = 1, id2 = 2, id3 = 3, id4 = 4, id5 = 5, id6 = 6 }). big_record_1_update(_Config) -> State = #big_record{}, call_big_record_1_update(?COUNT, State). call_big_record_1_update(0, _) -> #big_record{id1 = 0}; call_big_record_1_update(Number_test, State) -> call_big_record_1_update(Number_test - 1, State#big_record{id1 = Number_test}). %%%%%%%%%%%%%%%%%%%%%%%%%%% erl -S testing.erl {function, all, 0, 2}. {label,1}. {line,[{location,"testing_06_SUITE.erl",5}]}. {func_info,{atom,testing_06_SUITE},{atom,all},0}. {label,2}. {move,{literal,[big_record_1_update,big_record_2_update, big_record_3_update]}, {x,0}}. return. {function, big_record_1_update, 1, 4}. {label,3}. {line,[{location,"testing_06_SUITE.erl",18}]}. {func_info,{atom,testing_06_SUITE},{atom,big_record_1_update},1}. {label,4}. {move,{literal,{big_record,1,2,3,4,5,6}},{x,1}}. {move,{integer,100000000},{x,0}}. {call_only,2,{f,6}}. {function, call_big_record_1_update, 2, 6}. {label,5}. {line,[{location,"testing_06_SUITE.erl",21}]}. {func_info,{atom,testing_06_SUITE},{atom,call_big_record_1_update},2}. {label,6}. {test,is_eq_exact,{f,7},[{x,0},{integer,0}]}. {move,{literal,{big_record,0,2,3,4,5,6}},{x,0}}. return. %% This is interesting moment __________________________________________ {label,7}. {allocate_zero,1,2}. {line,[{location,"testing_06_SUITE.erl",24}]}. {gc_bif,'-',{f,0},2,[{x,0},{integer,1}],{x,2}}. {test,is_tuple,{f,8},[{x,1}]}. {test,test_arity,{f,8},[{x,1},7]}. {get_tuple_element,{x,1},0,{x,3}}. {test,is_eq_exact,{f,8},[{x,3},{atom,big_record}]}. {move,{x,2},{y,0}}. {move,{x,0},{x,2}}. {move,{integer,2},{x,0}}. {line,[{location,"testing_06_SUITE.erl",24}]}. {call_ext,3,{extfunc,erlang,setelement,3}}. {move,{x,0},{x,1}}. {move,{y,0},{x,0}}. {call_last,2,{f,6},1}. __________________________________________ {label,8}. {move,{literal,{badrecord,big_record}},{x,0}}. {line,[{location,"testing_06_SUITE.erl",24}]}. {call_ext,1,{extfunc,erlang,error,1}}. %%%%%%%%%%%%%%%%%%%%%%%%%%% erl -S +strict_record_updates testing.erl {function, all, 0, 2}. {label,1}. {line,[{location,"testing_06_SUITE.erl",5}]}. {func_info,{atom,testing_06_SUITE},{atom,all},0}. {label,2}. {move,{literal,[big_record_1_update,big_record_2_update, big_record_3_update]}, {x,0}}. return. {function, big_record_1_update, 1, 4}. {label,3}. {line,[{location,"testing_06_SUITE.erl",18}]}. {func_info,{atom,testing_06_SUITE},{atom,big_record_1_update},1}. {label,4}. {move,{literal,{big_record,1,2,3,4,5,6}},{x,1}}. {move,{integer,100000000},{x,0}}. {call_only,2,{f,6}}. {function, call_big_record_1_update, 2, 6}. {label,5}. {line,[{location,"testing_06_SUITE.erl",21}]}. {func_info,{atom,testing_06_SUITE},{atom,call_big_record_1_update},2}. {label,6}. {test,is_eq_exact,{f,7},[{x,0},{integer,0}]}. {move,{literal,{big_record,0,2,3,4,5,6}},{x,0}}. return. %% This is interesting moment __________________________________________ {label,7}. {line,[{location,"testing_06_SUITE.erl",24}]}. {gc_bif,'-',{f,0},2,[{x,0},{integer,1}],{x,2}}. {test,is_tuple,{f,8},[{x,1}]}. {test,test_arity,{f,8},[{x,1},7]}. {get_tuple_element,{x,1},0,{x,3}}. {get_tuple_element,{x,1},2,{x,4}}. {get_tuple_element,{x,1},3,{x,5}}. {get_tuple_element,{x,1},4,{x,6}}. {get_tuple_element,{x,1},5,{x,7}}. {get_tuple_element,{x,1},6,{x,8}}. {test,is_eq_exact,{f,8},[{x,3},{atom,big_record}]}. {test_heap,8,9}. {put_tuple,7,{x,1}}. {put,{atom,big_record}}. {put,{x,0}}. {put,{x,4}}. {put,{x,5}}. {put,{x,6}}. {put,{x,7}}. {put,{x,8}}. {move,{x,2},{x,0}}. {call_only,2,{f,6}}. __________________________________________ {label,8}. {move,{literal,{badrecord,big_record}},{x,0}}. {line,[{location,"testing_06_SUITE.erl",24}]}. {call_ext,1,{extfunc,erlang,error,1}}. -------------- next part -------------- An HTML attachment was scrubbed... URL: From z_m@REDACTED Tue Nov 17 12:12:46 2015 From: z_m@REDACTED (Zdenek Mejzlik) Date: Tue, 17 Nov 2015 12:12:46 +0100 Subject: [erlang-questions] Cowboy tutorial Message-ID: <564B0BAE.2090301@volny.cz> Hi, I'm new to Cowboy even new to Erlang. I'm testing example on http://ninenines.eu/docs/en/cowboy/1.0/guide/getting_started/ but I'm not successful. I have followed the steps carefully. Only suspicious place was paragraph starting with "Modifying the application resource file, |src/hello_erlang.app.src"| Make didn't created this file. So I created it by myself. Everything goes well and outputs look reasonably. But at the end when I enter http://localhost:8080 into my browser I got "Unable to connect" error. I use Erlang 18.1 on Linux Mint 17.2 64bit. Any advice? Thank you. Z. From essen@REDACTED Tue Nov 17 13:58:03 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 17 Nov 2015 13:58:03 +0100 Subject: [erlang-questions] Cowboy tutorial In-Reply-To: <564B0BAE.2090301@volny.cz> References: <564B0BAE.2090301@volny.cz> Message-ID: <564B245B.90807@ninenines.eu> Hello, On 11/17/2015 12:12 PM, Zdenek Mejzlik wrote: > Hi, > > I'm new to Cowboy even new to Erlang. > > I'm testing example on > http://ninenines.eu/docs/en/cowboy/1.0/guide/getting_started/ > > but I'm not successful. > > I have followed the steps carefully. Only suspicious place was paragraph > starting with "Modifying the application resource file, > |src/hello_erlang.app.src"| > Make didn't created this file. So I created it by myself. You don't need this file anymore. I will open a ticket to update the documentation. > Everything goes well and outputs look reasonably. But at the end when I > enter http://localhost:8080 into my browser I got "Unable to connect" > error. Please push the project somewhere so I can investigate. > I use Erlang 18.1 on Linux Mint 17.2 64bit. > > Any advice? Thank you. > > Z. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From z_m@REDACTED Tue Nov 17 18:40:32 2015 From: z_m@REDACTED (Zdenek Mejzlik) Date: Tue, 17 Nov 2015 18:40:32 +0100 Subject: [erlang-questions] Cowboy tutorial In-Reply-To: <564B245B.90807@ninenines.eu> References: <564B0BAE.2090301@volny.cz> <564B245B.90807@ninenines.eu> Message-ID: <564B6690.7060601@volny.cz> Hi Loic, Vladimir. I found an error in my procedure. In the last step of the tutorial I started the release in a bad way. Noob's mistake I did repeatedly. Now it works. Thank you very much for your efforts to help. Z. On 11/17/2015 01:58 PM, Lo?c Hoguin wrote: > Hello, > > On 11/17/2015 12:12 PM, Zdenek Mejzlik wrote: >> Hi, >> >> I'm new to Cowboy even new to Erlang. >> >> I'm testing example on >> http://ninenines.eu/docs/en/cowboy/1.0/guide/getting_started/ >> >> but I'm not successful. >> >> I have followed the steps carefully. Only suspicious place was paragraph >> starting with "Modifying the application resource file, >> |src/hello_erlang.app.src"| >> Make didn't created this file. So I created it by myself. > > You don't need this file anymore. I will open a ticket to update the > documentation. > >> Everything goes well and outputs look reasonably. But at the end when I >> enter http://localhost:8080 into my browser I got "Unable to connect" >> error. > > Please push the project somewhere so I can investigate. > >> I use Erlang 18.1 on Linux Mint 17.2 64bit. >> >> Any advice? Thank you. >> >> Z. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From povetkin.k@REDACTED Wed Nov 18 18:22:14 2015 From: povetkin.k@REDACTED (Constantine Povietkin) Date: Wed, 18 Nov 2015 19:22:14 +0200 Subject: [erlang-questions] Rebar dependency tracking Message-ID: <20151118192214.21d0e4e9@kappac.home.local> Hi all! I have an application that depends on other applications with its own dependencies. A / \ B C /\ C D Rebar get-deps command fetches dependencies to the A deps folder. As a result dependency B could not be compiled because of missing dependency. I use rebar 2.5.0. rebar.config: {deps, [ {C, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, {B, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, ]}. What am I doing wrong? Or where can I read about dependency tracking more detail? Thanks in advance. From dmkolesnikov@REDACTED Wed Nov 18 19:34:47 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 18 Nov 2015 20:34:47 +0200 Subject: [erlang-questions] Rebar dependency tracking In-Reply-To: <20151118192214.21d0e4e9@kappac.home.local> References: <20151118192214.21d0e4e9@kappac.home.local> Message-ID: <3CD4492F-A05D-47A9-B493-766773E7734B@gmail.com> Hi, It should work. I might assume that B depends on diff version of C Can you share error message with us? Best Regards, Dmitry > On Nov 18, 2015, at 7:22 PM, Constantine Povietkin wrote: > > Hi all! > > I have an application that depends on other applications with its own > dependencies. > > A > / \ > B C > /\ > C D > > Rebar get-deps command fetches dependencies to the A deps folder. As a > result dependency B could not be compiled because of missing dependency. > > I use rebar 2.5.0. > > rebar.config: > > {deps, [ > {C, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, > {B, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, > ]}. > > > What am I doing wrong? Or where can I read about dependency tracking > more detail? > > Thanks in advance. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From povetkin.k@REDACTED Wed Nov 18 19:58:30 2015 From: povetkin.k@REDACTED (Constantine Povietkin) Date: Wed, 18 Nov 2015 20:58:30 +0200 Subject: [erlang-questions] Rebar dependency tracking In-Reply-To: <3CD4492F-A05D-47A9-B493-766773E7734B@gmail.com> References: <20151118192214.21d0e4e9@kappac.home.local> <3CD4492F-A05D-47A9-B493-766773E7734B@gmail.com> Message-ID: <20151118205830.05777d04@kappac.home.local> ? Wed, 18 Nov 2015 20:34:47 +0200 Dmitry Kolesnikov ?????: No, B depends on the version as B does. Message: could find file or directory. And "deps" folder for B is empty. > Hi, > > It should work. I might assume that B depends on diff version of C > Can you share error message with us? > > Best Regards, Dmitry > > > > On Nov 18, 2015, at 7:22 PM, Constantine Povietkin > > wrote: > > > > Hi all! > > > > I have an application that depends on other applications with its > > own dependencies. > > > > A > > / \ > > B C > > /\ > > C D > > > > Rebar get-deps command fetches dependencies to the A deps folder. > > As a result dependency B could not be compiled because of missing > > dependency. > > > > I use rebar 2.5.0. > > > > rebar.config: > > > > {deps, [ > > {C, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, > > {B, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, > > ]}. > > > > > > What am I doing wrong? Or where can I read about dependency tracking > > more detail? > > > > Thanks in advance. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > From dmkolesnikov@REDACTED Wed Nov 18 20:29:17 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 18 Nov 2015 21:29:17 +0200 Subject: [erlang-questions] Rebar dependency tracking In-Reply-To: <20151118205830.05777d04@kappac.home.local> References: <20151118192214.21d0e4e9@kappac.home.local> <3CD4492F-A05D-47A9-B493-766773E7734B@gmail.com> <20151118205830.05777d04@kappac.home.local> Message-ID: <45FC4326-BD48-4B6E-8FF2-C590E646FF62@gmail.com> There should be a flat structure of deps unless you mess-up with rebar.config Do you have any {deps_dir, ?}. directive in rebar.config of B,C, or A projects? - Dmitry > On Nov 18, 2015, at 8:58 PM, Constantine Povietkin wrote: > > ? Wed, 18 Nov 2015 20:34:47 +0200 > Dmitry Kolesnikov ?????: > > No, B depends on the version as B does. > > Message: could find file or directory. > > And "deps" folder for B is empty. > >> Hi, >> >> It should work. I might assume that B depends on diff version of C >> Can you share error message with us? >> >> Best Regards, Dmitry >> >> >>> On Nov 18, 2015, at 7:22 PM, Constantine Povietkin >>> wrote: >>> >>> Hi all! >>> >>> I have an application that depends on other applications with its >>> own dependencies. >>> >>> A >>> / \ >>> B C >>> /\ >>> C D >>> >>> Rebar get-deps command fetches dependencies to the A deps folder. >>> As a result dependency B could not be compiled because of missing >>> dependency. >>> >>> I use rebar 2.5.0. >>> >>> rebar.config: >>> >>> {deps, [ >>> {C, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, >>> {B, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, >>> ]}. >>> >>> >>> What am I doing wrong? Or where can I read about dependency tracking >>> more detail? >>> >>> Thanks in advance. >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > From povetkin.k@REDACTED Wed Nov 18 21:02:27 2015 From: povetkin.k@REDACTED (Constantine Povietkin) Date: Wed, 18 Nov 2015 22:02:27 +0200 Subject: [erlang-questions] Rebar dependency tracking In-Reply-To: <45FC4326-BD48-4B6E-8FF2-C590E646FF62@gmail.com> References: <20151118192214.21d0e4e9@kappac.home.local> <3CD4492F-A05D-47A9-B493-766773E7734B@gmail.com> <20151118205830.05777d04@kappac.home.local> <45FC4326-BD48-4B6E-8FF2-C590E646FF62@gmail.com> Message-ID: <20151118220227.5621ddf5@kappac.home.local> ? Wed, 18 Nov 2015 21:29:17 +0200 Dmitry Kolesnikov ?????: No, I don't. sub_dirs and deps only. > There should be a flat structure of deps unless you mess-up with > rebar.config Do you have any {deps_dir, ?}. directive in rebar.config > of B,C, or A projects? > > - Dmitry > > > On Nov 18, 2015, at 8:58 PM, Constantine Povietkin > > wrote: > > > > ? Wed, 18 Nov 2015 20:34:47 +0200 > > Dmitry Kolesnikov ?????: > > > > No, B depends on the version as B does. > > > > Message: could find file or directory. > > > > And "deps" folder for B is empty. > > > >> Hi, > >> > >> It should work. I might assume that B depends on diff version of C > >> Can you share error message with us? > >> > >> Best Regards, Dmitry > >> > >> > >>> On Nov 18, 2015, at 7:22 PM, Constantine Povietkin > >>> wrote: > >>> > >>> Hi all! > >>> > >>> I have an application that depends on other applications with its > >>> own dependencies. > >>> > >>> A > >>> / \ > >>> B C > >>> /\ > >>> C D > >>> > >>> Rebar get-deps command fetches dependencies to the A deps folder. > >>> As a result dependency B could not be compiled because of missing > >>> dependency. > >>> > >>> I use rebar 2.5.0. > >>> > >>> rebar.config: > >>> > >>> {deps, [ > >>> {C, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, > >>> {B, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, > >>> ]}. > >>> > >>> > >>> What am I doing wrong? Or where can I read about dependency > >>> tracking more detail? > >>> > >>> Thanks in advance. > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >> > > > From kennethlakin@REDACTED Thu Nov 19 05:46:59 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 18 Nov 2015 20:46:59 -0800 Subject: [erlang-questions] Rebar dependency tracking In-Reply-To: <20151118192214.21d0e4e9@kappac.home.local> References: <20151118192214.21d0e4e9@kappac.home.local> Message-ID: <564D5443.4090703@gmail.com> On 11/18/2015 09:22 AM, Constantine Povietkin wrote: > Hi all! > > I have an application that depends on other applications with its own > dependencies. > Rebar get-deps command fetches dependencies to the A deps folder. As a > result dependency B could not be compiled because > of missing dependency. I'm not *exactly* sure what you're asking, but does the way I'm doing it in my rebar 2.x-based project [0] do what you're trying to do? If it doesn't, then I pre-emptively apologize for the noise. My project looks like: \rebar.conf <--- contains *only* a sub_dirs directive \deps\ <--- *all* fetched deps go in here \apps\mta_fetcher\rebar.conf \apps\mta_fetcher_database\rebar.conf [0] https://github.com/kennethlakin/mta-fetcher -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From be.dmitry@REDACTED Thu Nov 19 07:48:22 2015 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Thu, 19 Nov 2015 17:48:22 +1100 Subject: [erlang-questions] Rebar dependency tracking In-Reply-To: <20151118192214.21d0e4e9@kappac.home.local> References: <20151118192214.21d0e4e9@kappac.home.local> Message-ID: <3A0A0EEA-1002-484A-99F4-4A38F219D7C4@gmail.com> Applications in erlang represent a flat list, so there may be only one version of an application used in runtime. Rebar expects all your dependencies to reside in the ?top-directory?/deps after rebar get-deps. The dependencies would use this exact folder to look for their own dependencies, so there would be no ./deps/*/deps directories. -- Best wishes, Dmitry Belyaev On 19 November 2015 4:22:14 AM AEDT, Constantine Povietkin wrote: >Hi all! > >I have an application that depends on other applications with its own >dependencies. > > A > / \ > B C > /\ > C D > >Rebar get-deps command fetches dependencies to the A deps folder. As a >result dependency B could not be compiled because of missing >dependency. > >I use rebar 2.5.0. > >rebar.config: > >{deps, [ > {C, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, > {B, ".*", {git, "git_path_to_dependecy", {tag, "v1.0.0"}}}, >]}. > > >What am I doing wrong? Or where can I read about dependency tracking >more detail? > >Thanks in advance. >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Thu Nov 19 13:36:21 2015 From: roger@REDACTED (Roger Lipscombe) Date: Thu, 19 Nov 2015 12:36:21 +0000 Subject: [erlang-questions] Mocks and explicit contracts Message-ID: In http://blog.plataformatec.com.br/2015/10/mocks-and-explicit-contracts/, Jos? Valim suggests a way to use dependency injection to more easily unit test Elixir code. However, it doesn't translate to Erlang... In one example, he stores the dependency (as a module) in the application environment and then defines a private function to retrieve it: defp twitter_api do Application.get_env(:my_app, :twitter_api) end Then it can be called as: twitter_api.get_username(username) This doesn't translate to Erlang (or, rather, it's unwieldy), because we can't do the following: twitter_api() -> application:get_env(my_app, twitter_api, default_twitter_api). show(Username) -> % syntax error before: ':' twitter_api():get_username(Username) etc. We need more brackets: (twitter_api()):get_username(Username) This gets a bit ugly, though less so with macros: -define(TWITTER_API, % note the extra brackets: (application:get_env(my_app, twitter_api, default_twitter_api))). show(Username) -> ?TWITTER_API:get_username(Username). He goes on to show another form of DI which takes advantage of the fact that Elixir allows inline modules, Again, we can't do this in Erlang. So: - Is Jose's approach applicable to Erlang at all? - Are there idiomatic ways to do something like this in Erlang? I know about (and use) 'meck', which is awesome, but has the disadvantage that it mocks (verb) globally, which limits running tests in parallel. It can also be quite slow, because it compiles the mocks on the fly. Jos?'s approach uses mocks (noun) locally. From schneider@REDACTED Thu Nov 19 14:37:32 2015 From: schneider@REDACTED (Frans Schneider) Date: Thu, 19 Nov 2015 14:37:32 +0100 Subject: [erlang-questions] How to use crypto:hmac_init() and friends? Message-ID: <564DD09C.2040704@xs4all.nl> Dear list, I have to create the HMAC of a message by including data at several points in my code as shown below: ... Mac_ctxt_init = crypto:hmac_init(sha256, Mac_key), Mac_ctxt = case Version of 3 -> Mac_ctxt1 = crypto:hmac_update(Mac_ctxt_init, Dhi_pub), crypto:hmac_update(Mac_ctxt1, Dhir_pub); 2 -> Mac_ctxt_init end, ... ... ... LMac_ctxt = crypto:hmac_update(Mac_ctxt, <>), LMac = crypto:hmac_final_n(LMac_ctxt, 8), ... This will always return <<>>! It already starts with the hmac_init() which returns <<>>. Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false] Eshell V7.1 (abort with ^G) 1> K = <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68,39,22,88,62,219,15,114,22,236,68,226,141,39,69,212,143,123>>. <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68, 39,22,88,62,219,15,114,22,236,68,226,141,39,69,...>> 2> crypto:hmac_init(sha256, K). <<>> 3> Using crypto:hmac(sha256, K, Data, 8) works fine, but is not usable. Any idea what goes wrong or what I am doing wrong? TIA, Frans From sverker.eriksson@REDACTED Thu Nov 19 15:18:42 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 19 Nov 2015 15:18:42 +0100 Subject: [erlang-questions] How to use crypto:hmac_init() and friends? In-Reply-To: <564DD09C.2040704@xs4all.nl> References: <564DD09C.2040704@xs4all.nl> Message-ID: <564DDA42.7040707@ericsson.com> On 11/19/2015 02:37 PM, Frans Schneider wrote: > Dear list, > > I have to create the HMAC of a message by including data at several > points in my code as shown below: > > ... > Mac_ctxt_init = crypto:hmac_init(sha256, Mac_key), > Mac_ctxt = case Version of > 3 -> > Mac_ctxt1 = crypto:hmac_update(Mac_ctxt_init, > Dhi_pub), > crypto:hmac_update(Mac_ctxt1, Dhir_pub); > 2 -> > Mac_ctxt_init > end, > ... > ... > ... > LMac_ctxt = crypto:hmac_update(Mac_ctxt, < Serialized/binary>>), > LMac = crypto:hmac_final_n(LMac_ctxt, 8), > ... > > This will always return <<>>! It already starts with the hmac_init() > which returns <<>>. > > Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] > [async-threads:10] [hipe] [kernel-poll:false] > > Eshell V7.1 (abort with ^G) > 1> K = > <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68,39,22,88,62,219,15,114,22,236,68,226,141,39,69,212,143,123>>. > <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68, > 39,22,88,62,219,15,114,22,236,68,226,141,39,69,...>> > 2> crypto:hmac_init(sha256, K). > <<>> > 3> > > Using crypto:hmac(sha256, K, Data, 8) works fine, but is not usable. > > Any idea what goes wrong or what I am doing wrong? > Nothing is wrong (ugly maybe, but not wrong). That empty binary is not really empty. It's a magic binary! It's referring some internal emulator state (the hmac context in this case). Just pass it along to hmac_update and hmac_final and you will get a nice regular binary in the end. The magic binary looks and behaves like an empty binary. It will even lose its magic powers and turn into a regular binary if serialized with term_to_binary. /Sverker, Erlang/OTP From be.dmitry@REDACTED Thu Nov 19 15:27:48 2015 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Fri, 20 Nov 2015 01:27:48 +1100 Subject: [erlang-questions] How to use crypto:hmac_init() and friends? In-Reply-To: <564DDA42.7040707@ericsson.com> References: <564DD09C.2040704@xs4all.nl> <564DDA42.7040707@ericsson.com> Message-ID: Wouldn't it make sense to introduce a separate data type for these custom data? Or even allow to define new custom types in nifs or drivers with serialization and comparison rules. -- Best wishes, Dmitry Belyaev On 20 November 2015 1:18:42 AM AEDT, Sverker Eriksson wrote: >On 11/19/2015 02:37 PM, Frans Schneider wrote: >> Dear list, >> >> I have to create the HMAC of a message by including data at several >> points in my code as shown below: >> >> ... >> Mac_ctxt_init = crypto:hmac_init(sha256, Mac_key), >> Mac_ctxt = case Version of >> 3 -> >> Mac_ctxt1 = crypto:hmac_update(Mac_ctxt_init, >> Dhi_pub), >> crypto:hmac_update(Mac_ctxt1, Dhir_pub); >> 2 -> >> Mac_ctxt_init >> end, >> ... >> ... >> ... >> LMac_ctxt = crypto:hmac_update(Mac_ctxt, <> Serialized/binary>>), >> LMac = crypto:hmac_final_n(LMac_ctxt, 8), >> ... >> >> This will always return <<>>! It already starts with the hmac_init() >> which returns <<>>. >> >> Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] >> [async-threads:10] [hipe] [kernel-poll:false] >> >> Eshell V7.1 (abort with ^G) >> 1> K = >> ><<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68,39,22,88,62,219,15,114,22,236,68,226,141,39,69,212,143,123>>. >> <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68, >> 39,22,88,62,219,15,114,22,236,68,226,141,39,69,...>> >> 2> crypto:hmac_init(sha256, K). >> <<>> >> 3> >> >> Using crypto:hmac(sha256, K, Data, 8) works fine, but is not usable. >> >> Any idea what goes wrong or what I am doing wrong? >> >Nothing is wrong (ugly maybe, but not wrong). > >That empty binary is not really empty. It's a magic binary! >It's referring some internal emulator state (the hmac context in this >case). > >Just pass it along to hmac_update and hmac_final and you will get a >nice >regular binary in the end. > >The magic binary looks and behaves like an empty binary. It will even >lose its magic powers >and turn into a regular binary if serialized with term_to_binary. > > >/Sverker, Erlang/OTP > > > > > > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Thu Nov 19 15:34:33 2015 From: schneider@REDACTED (Frans Schneider) Date: Thu, 19 Nov 2015 15:34:33 +0100 Subject: [erlang-questions] How to use crypto:hmac_init() and friends? In-Reply-To: <564DDA42.7040707@ericsson.com> References: <564DD09C.2040704@xs4all.nl> <564DDA42.7040707@ericsson.com> Message-ID: <564DDDF9.1030704@xs4all.nl> Ok, thanks. It does work indeed. But what do you mean with 'Ugly'"? /Frans On 11/19/2015 03:18 PM, Sverker Eriksson wrote: > On 11/19/2015 02:37 PM, Frans Schneider wrote: >> Dear list, >> >> I have to create the HMAC of a message by including data at several >> points in my code as shown below: >> >> ... >> Mac_ctxt_init = crypto:hmac_init(sha256, Mac_key), >> Mac_ctxt = case Version of >> 3 -> >> Mac_ctxt1 = crypto:hmac_update(Mac_ctxt_init, >> Dhi_pub), >> crypto:hmac_update(Mac_ctxt1, Dhir_pub); >> 2 -> >> Mac_ctxt_init >> end, >> ... >> ... >> ... >> LMac_ctxt = crypto:hmac_update(Mac_ctxt, <> Serialized/binary>>), >> LMac = crypto:hmac_final_n(LMac_ctxt, 8), >> ... >> >> This will always return <<>>! It already starts with the hmac_init() >> which returns <<>>. >> >> Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] >> [async-threads:10] [hipe] [kernel-poll:false] >> >> Eshell V7.1 (abort with ^G) >> 1> K = >> <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68,39,22,88,62,219,15,114,22,236,68,226,141,39,69,212,143,123>>. >> <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68, >> 39,22,88,62,219,15,114,22,236,68,226,141,39,69,...>> >> 2> crypto:hmac_init(sha256, K). >> <<>> >> 3> >> >> Using crypto:hmac(sha256, K, Data, 8) works fine, but is not usable. >> >> Any idea what goes wrong or what I am doing wrong? >> > Nothing is wrong (ugly maybe, but not wrong). > > That empty binary is not really empty. It's a magic binary! > It's referring some internal emulator state (the hmac context in this > case). > > Just pass it along to hmac_update and hmac_final and you will get a > nice regular binary in the end. > > The magic binary looks and behaves like an empty binary. It will even > lose its magic powers > and turn into a regular binary if serialized with term_to_binary. > > > /Sverker, Erlang/OTP > > > > > > From sverker.eriksson@REDACTED Thu Nov 19 16:00:10 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 19 Nov 2015 16:00:10 +0100 Subject: [erlang-questions] How to use crypto:hmac_init() and friends? In-Reply-To: References: <564DD09C.2040704@xs4all.nl> <564DDA42.7040707@ericsson.com> Message-ID: <564DE3FA.9010700@ericsson.com> On 11/19/2015 03:27 PM, Dmitry Belyaev wrote: > Wouldn't it make sense to introduce a separate data type for these > custom data? Or even allow to define new custom types in nifs or > drivers with serialization and comparison rules. > -- > Yes, that would make sense. Introducing a new data type in the language is quite a big operation though. These empty magic binaries was sort of a last minute solution to give NIF designers some way to return "safe pointers" to Erlang. /Sverker, Erlang/OTP -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Thu Nov 19 16:06:05 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 19 Nov 2015 16:06:05 +0100 Subject: [erlang-questions] How to use crypto:hmac_init() and friends? In-Reply-To: <564DDDF9.1030704@xs4all.nl> References: <564DD09C.2040704@xs4all.nl> <564DDA42.7040707@ericsson.com> <564DDDF9.1030704@xs4all.nl> Message-ID: <564DE55D.80506@ericsson.com> That was self-criticism. I was referring to the empty binary that's not really empty. /Sverker, Erlang/OTP On 11/19/2015 03:34 PM, Frans Schneider wrote: > Ok, thanks. It does work indeed. > But what do you mean with 'Ugly'"? > > /Frans > > On 11/19/2015 03:18 PM, Sverker Eriksson wrote: >> On 11/19/2015 02:37 PM, Frans Schneider wrote: >>> Dear list, >>> >>> I have to create the HMAC of a message by including data at several >>> points in my code as shown below: >>> >>> ... >>> Mac_ctxt_init = crypto:hmac_init(sha256, Mac_key), >>> Mac_ctxt = case Version of >>> 3 -> >>> Mac_ctxt1 = crypto:hmac_update(Mac_ctxt_init, >>> Dhi_pub), >>> crypto:hmac_update(Mac_ctxt1, Dhir_pub); >>> 2 -> >>> Mac_ctxt_init >>> end, >>> ... >>> ... >>> ... >>> LMac_ctxt = crypto:hmac_update(Mac_ctxt, <>> Serialized/binary>>), >>> LMac = crypto:hmac_final_n(LMac_ctxt, 8), >>> ... >>> >>> This will always return <<>>! It already starts with the hmac_init() >>> which returns <<>>. >>> >>> Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] >>> [async-threads:10] [hipe] [kernel-poll:false] >>> >>> Eshell V7.1 (abort with ^G) >>> 1> K = >>> <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68,39,22,88,62,219,15,114,22,236,68,226,141,39,69,212,143,123>>. >>> <<9,138,157,154,221,37,75,104,28,207,127,232,193,207,68, >>> 39,22,88,62,219,15,114,22,236,68,226,141,39,69,...>> >>> 2> crypto:hmac_init(sha256, K). >>> <<>> >>> 3> >>> >>> Using crypto:hmac(sha256, K, Data, 8) works fine, but is not usable. >>> >>> Any idea what goes wrong or what I am doing wrong? >>> >> Nothing is wrong (ugly maybe, but not wrong). >> >> That empty binary is not really empty. It's a magic binary! >> It's referring some internal emulator state (the hmac context in this >> case). >> >> Just pass it along to hmac_update and hmac_final and you will get a >> nice regular binary in the end. >> >> The magic binary looks and behaves like an empty binary. It will even >> lose its magic powers >> and turn into a regular binary if serialized with term_to_binary. >> >> >> /Sverker, Erlang/OTP >> >> >> >> >> >> > > From Ola.Backstrom@REDACTED Thu Nov 19 16:12:30 2015 From: Ola.Backstrom@REDACTED (=?utf-8?B?T2xhIELDpGNrc3Ryw7Zt?=) Date: Thu, 19 Nov 2015 15:12:30 +0000 Subject: [erlang-questions] Mocks and explicit contracts In-Reply-To: References: Message-ID: I don't see that how one can get away with running independent tests in parallel with mocks or doing the get_env-thing. The latter do work, though, IFF all concerned tests are just expecting the same thing from the called function... but most likely you have some tests that would check different scenarios. One test might check that the System Under Test behaves well if twitter does a positive response while another might check that the SUT does not fail in a catastrophic way if the twitter call gives garbage as response. From a performance and code readability perspective I'd rather have normal module calls like: some_twitter_api:get_username(Username) (and having to use the excellent meck mock library) instead of something like ?TWITTER_API:get_username(Username) Or (twitter_api()):get_username(Username) In particular, having every module-prefixed call doing an application:get_env-call underneath does not feel very erlangish. And it doesn't really solve the running-the-test-suite-concurrently issue, in the more general case where tests are expecting different return values from a function. Regarding mocks (meck) being slow, it is the meck:new that costs time. That function swap the module with a runtime generated module. (meck:unload is similar) If you, for instance, make a wrapper around meck that calls meck:new and meck:unload once in the test suite and uses meck:delete to remove mocks between individual test cases you can increase test suite execution speed a lot. /Ola -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Roger Lipscombe Sent: den 19 november 2015 13:36 To: erlang-questions@REDACTED Subject: [erlang-questions] Mocks and explicit contracts In http://blog.plataformatec.com.br/2015/10/mocks-and-explicit-contracts/, Jos? Valim suggests a way to use dependency injection to more easily unit test Elixir code. However, it doesn't translate to Erlang... In one example, he stores the dependency (as a module) in the application environment and then defines a private function to retrieve it: defp twitter_api do Application.get_env(:my_app, :twitter_api) end Then it can be called as: twitter_api.get_username(username) This doesn't translate to Erlang (or, rather, it's unwieldy), because we can't do the following: twitter_api() -> application:get_env(my_app, twitter_api, default_twitter_api). show(Username) -> % syntax error before: ':' twitter_api():get_username(Username) etc. We need more brackets: (twitter_api()):get_username(Username) This gets a bit ugly, though less so with macros: -define(TWITTER_API, % note the extra brackets: (application:get_env(my_app, twitter_api, default_twitter_api))). show(Username) -> ?TWITTER_API:get_username(Username). He goes on to show another form of DI which takes advantage of the fact that Elixir allows inline modules, Again, we can't do this in Erlang. So: - Is Jose's approach applicable to Erlang at all? - Are there idiomatic ways to do something like this in Erlang? I know about (and use) 'meck', which is awesome, but has the disadvantage that it mocks (verb) globally, which limits running tests in parallel. It can also be quite slow, because it compiles the mocks on the fly. Jos?'s approach uses mocks (noun) locally. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From jose.valim@REDACTED Thu Nov 19 16:39:09 2015 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Thu, 19 Nov 2015 23:39:09 +0800 Subject: [erlang-questions] Mocks and explicit contracts In-Reply-To: References: Message-ID: > > The latter do work, though, IFF all concerned tests are just expecting the > same thing from the called function... but most likely you have some tests > that would check different scenarios. One test might check that the System > Under Test behaves well if twitter does a positive response while another > might check that the SUT does not fail in a catastrophic way if the > twitter call gives garbage as response. > Yes. One solution to this problem is to parameterize the different responses under the same system. For example, the username request for "josevalim" would work but fail for "josevalim-fail". It is a very simple solution but it can get unwieldy for complex systems where you need to test many scenarios. In any case, I wouldn't discard this approach quickly, as it fits simple problems quite well. A more complex solution is to have an ownership pool where you could say: "if this pid asks for this data, return this value". The benefit of such system is that, if a pid has a given key reassigned without first "deallocating" it, you certainly have a concurrency issue in your tests where two tests need the same resource. We are building such system to allow us to run tests concurrently even if they depend on a common external resource like a database. We replace the regular database pool by the ownership pool and run tests concurrently by wrapping each connection in a transaction (works for SQL ones) or by running different database instances. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Nov 19 16:44:30 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 19 Nov 2015 16:44:30 +0100 Subject: [erlang-questions] Mocks and explicit contracts In-Reply-To: References: Message-ID: Another solution again is to run all the tests in parallel and then afterwards you make a reconstruction in which you ask the question "Can I reconstruct a trace from the observations I made?" That solution is sometimes easier to pull off, though the reconstruction can be computationally expensive in the general case. This is, more or less, what you would do in a property based testing solution. On Thu, Nov 19, 2015 at 4:39 PM, Jos? Valim wrote: > The latter do work, though, IFF all concerned tests are just expecting the >> same thing from the called function... but most likely you have some tests >> that would check different scenarios. One test might check that the System >> Under Test behaves well if twitter does a positive response while another >> might check that the SUT does not fail in a catastrophic way if the >> twitter call gives garbage as response. >> > > Yes. One solution to this problem is to parameterize the different > responses under the same system. For example, the username request for > "josevalim" would work but fail for "josevalim-fail". It is a very simple > solution but it can get unwieldy for complex systems where you need to test > many scenarios. In any case, I wouldn't discard this approach quickly, as > it fits simple problems quite well. > > A more complex solution is to have an ownership pool where you could say: > "if this pid asks for this data, return this value". The benefit of such > system is that, if a pid has a given key reassigned without first > "deallocating" it, you certainly have a concurrency issue in your tests > where two tests need the same resource. We are building such system to > allow us to run tests concurrently even if they depend on a common external > resource like a database. We replace the regular database pool by the > ownership pool and run tests concurrently by wrapping each connection in a > transaction (works for SQL ones) or by running different database instances. > > > *Jos? Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Director of R&D > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Nov 19 16:53:06 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 19 Nov 2015 10:53:06 -0500 Subject: [erlang-questions] Mocks and explicit contracts In-Reply-To: References: Message-ID: <20151119155305.GB40671@fhebert-ltm1> On 11/19, Roger Lipscombe wrote: >In http://blog.plataformatec.com.br/2015/10/mocks-and-explicit-contracts/, >Jos? Valim suggests a way to use dependency injection to more easily >unit test Elixir code. > >However, it doesn't translate to Erlang... > >In one example, he stores the dependency (as a module) in the >application environment and then defines a private function to >retrieve it: > > defp twitter_api do > Application.get_env(:my_app, :twitter_api) > end > >Then it can be called as: > > twitter_api.get_username(username) > Dear god is that a possibility? I'm pretty annoyed to begin with, at the prospect of a seeming module name. I figure that's the Erlanger in me being confused by Elixir using Uppercased module names and lowercased function/variable names there. The insidious thing about this practice is that every function call you make to the twitter API now includes: - an access to a shared ETS table, acquiring a read lock - extracting the value from ETS and copying it to your process - an access to a shared ETS table, releasing a read lock - a lookup to make a dynamic fully-qualified function call The interesting thing then, is that how fast you can call your function in production now depends on how many concurrent calls exist and how much flexibility was desired in tests. That's not great. >- Is Jose's approach applicable to Erlang at all? >- Are there idiomatic ways to do something like this in Erlang? > >I know about (and use) 'meck', which is awesome, but has the >disadvantage that it mocks (verb) globally, which limits running tests >in parallel. It can also be quite slow, because it compiles the mocks >on the fly. Jos?'s approach uses mocks (noun) locally. First of all, Jos?'s approach also mocks globally in the Erlang context. It might be different using local modules, but if you enter Erlang, then it's automatically global. Would you consider, for that case, using instead: -ifndef(twitter_api). -define(twitter_api, default_twitter_api). -endif. show(Username) -> ?twitter_api:get_username(Username). Through build tools, you could define (here I'm using rebar3): {profiles, [ [{test, [{erl_opts, [{d, twitter_api, test_twitter_api}]}]}] ]}. You now get the flexibility you want, without mocks, without global data (although it is globally mocked), and without runtime cost, at the added visual overhead of a single '?' at the call-site. Then again, I'm not a super big fan of doing that kind of stuff. In the end, the questions I have to ask is: Q1: Am I going to need multiple modules there because it's a value I want to be configurable by users of a library at runtime? A1: if so, then application environment is not the right way, unless I plan on users only being able to configure one thing. If more than one library or OTP app relies on my library, they have to agree on the implementation. That sucks. The best way to do this form is to configure the module explicitly when starting the processes, and have it carry it around in state. This is, essentially, the safest way to concurrently allow multiple configurations at once. Q2: Am I needing multiple modules to test alternating configurations? A2: If so, consider the answer A1. If this is considered costly and the configuration is mostly static (i.e. windows vs. linux yielding different results), then consider the macros mentioned above. Q3: Am I needing to swap an implementation to make testing easier? A3: If so, use mocking. It requires no code change, and it's a much better idea (to me, anyway) to slow down test execution a bit than slowing down runtime execution because of my testing requirements. Talk about side-effects! If really mocking is unpractical, consider the macros. If someone sent me that 'globally configurable through app env dynamic module calls' in a code review, it would not make it to production. I would ask for this to be reviewed, because I think it's a lazy shortcut that has too many unintended consequences to be acceptable. I don't care for the syntax, it's just bad semantics, runtime properties, etc. Here's a short experiment: -module(mod). -compile(export_all). f(M) -> {timer:tc(?MODULE, ncalls, [100000, fun() -> mod:g() end]), timer:tc(?MODULE, ncalls, [100000, fun() -> M:g() end]), timer:tc(?MODULE, ncalls, [100000, fun() -> (application:get_env(app, val, M)):g() end])}. g() -> ok. ncalls(0, _) -> ok; ncalls(N, F) -> F(), ncalls(N-1, F). What's the runtime results for 100,000 calls? - mod:g(): 5827 ?s - M:g(): 7326 ?s - (application:get_env(app, val, M)):g(): 30770 ?s The solution chosen if the last one is taken is therefore agreeing that every one of your function calls to that API in production are going to be roughly 5x slower, *because* you wanted tests to run faster. It will also require code changes and variations that are otherwise not required with mocks. The cost of that application:get_env would be the same in Elixir I figure. It's just an inadequate solution to a problem that has been solved better, multiple times over. I personally prefer to pay the cost of slightly longer test runs personally, than having my users pay the cost themselves at run time. Regards, Fred. From povetkin.k@REDACTED Thu Nov 19 17:03:50 2015 From: povetkin.k@REDACTED (Constantine Povietkin) Date: Thu, 19 Nov 2015 18:03:50 +0200 Subject: [erlang-questions] Rebar dependency tracking In-Reply-To: <564D5443.4090703@gmail.com> References: <20151118192214.21d0e4e9@kappac.home.local> <564D5443.4090703@gmail.com> Message-ID: <20151119180350.7c6475f7@kappac.home.local> I'm trying to do the same way as you do in your project. ? Wed, 18 Nov 2015 20:46:59 -0800 Kenneth Lakin ?????: > On 11/18/2015 09:22 AM, Constantine Povietkin wrote: > > Hi all! > > > > I have an application that depends on other applications with its > > own dependencies. > > Rebar get-deps command fetches dependencies to the A deps folder. > > As a result dependency B could not be compiled because > > of missing dependency. > > I'm not *exactly* sure what you're asking, but does the way I'm doing > it in my rebar 2.x-based project [0] do what you're trying to do? If > it doesn't, then I pre-emptively apologize for the noise. > > My project looks like: > \rebar.conf <--- contains *only* a sub_dirs directive > \deps\ <--- *all* fetched deps go in here > \apps\mta_fetcher\rebar.conf > \apps\mta_fetcher_database\rebar.conf > > [0] https://github.com/kennethlakin/mta-fetcher > > From povetkin.k@REDACTED Thu Nov 19 17:06:18 2015 From: povetkin.k@REDACTED (Constantine Povietkin) Date: Thu, 19 Nov 2015 18:06:18 +0200 Subject: [erlang-questions] Rebar dependency tracking In-Reply-To: <3A0A0EEA-1002-484A-99F4-4A38F219D7C4@gmail.com> References: <20151118192214.21d0e4e9@kappac.home.local> <3A0A0EEA-1002-484A-99F4-4A38F219D7C4@gmail.com> Message-ID: <20151119180618.35845c1b@kappac.home.local> Great! Thank you a ton! The problem of my build process is relative paths in pre_hook method of one of the dependencies. ? Thu, 19 Nov 2015 17:48:22 +1100 Dmitry Belyaev ?????: > Applications in erlang represent a flat list, so there may be only > one version of an application used in runtime. Rebar expects all your > dependencies to reside in the ?top-directory?/deps after rebar > get-deps. The dependencies would use this exact folder to look for > their own dependencies, so there would be no ./deps/*/deps > directories. > > From sid5@REDACTED Thu Nov 19 17:17:21 2015 From: sid5@REDACTED (Sid Muller) Date: Thu, 19 Nov 2015 17:17:21 +0100 Subject: [erlang-questions] IP search optimization Message-ID: Hi, I was wondering if someone knowledgeable in IP/CIDR and bitstrings could give me some advice on how to proceed? I'm storing network address in an ets table, for example: 77.242.16.0/20 31.24.168.0/21 And then I want to check if an IP (31.24.171.44) belongs to any network address in the ets table (the table can get pretty large). The crux of the matter is that I am not sure which of the following routes to pursue because I only get the IP but CIDR is stored in ets with each network: 1. Should I store the network/CIDR in ets as tuples: {{77, 42, 6, 0}, 20} {{31, 24, 168, 0}, 21} and search for the IP as a tuple ({31, 24, 171, 44}) byte by byte? 2. Should I store the Network as a bitstring: <<77,242,1:4>> <<31,24,21:5>> And then search for the IP as a bitstring match? 3. Should I store the network/CIDR as binary: <<"77.242.16.0/20">> <<"31.24.168.0/21">> I think the best solution would be the #2 but this one, just like the others I don't know the CIDR ahead of time to be able to trim the CIDR off the IP and then do a simple key based match from ets. Any thoughts? Thank you From axling@REDACTED Thu Nov 19 17:19:34 2015 From: axling@REDACTED (Erik Axling) Date: Thu, 19 Nov 2015 17:19:34 +0100 Subject: [erlang-questions] fannerl, erlang bindings to FANN, the Fast Artificial Neural Network library Message-ID: Hello! I would like to promote an application that I have made called fannerl. It is erlang bindings to the FANN, Fast Artificial Neural Networks, library written by Steffen Nissen. To see more about FANN please check out: http://leenissen.dk/fann/wp/ Fannerl is a port driver that interfaces the libfanndouble version of FANN. A lot of the functionality is interfaced, not all though, please see the Github repo for more information. Github Link: https://github.com/axling/fannerl Docs: http://axling.github.io/fannerl/ Releases: https://github.com/axling/fannerl/releases Currently the only release is v1.0.0-alpha. The github repo contains examples of how to use fannerl and you can also read the docs there. Consider this in alpha mode as I have only tested this on a Mac OS X machine as of yet. I would much appreciate your help in testing this on other platforms. Please report any problems either directly to me, or even better use the issue tracker in github so more people can see the current issues. Note, Erlang/OTP version must be at least 17.0. FANN version must be at least 2.2.0. Best Regards Erik Axling -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Thu Nov 19 17:25:50 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 19 Nov 2015 18:25:50 +0200 Subject: [erlang-questions] IP search optimization In-Reply-To: References: Message-ID: Hi, I?ve did a similar but with persistent storage. * use aton function to map CIDR into integer space * use ordered_set ets table with CIDR as key then you can use ets:next(?) to map IP to CIDR Best Regards, Dmitry > On Nov 19, 2015, at 6:17 PM, Sid Muller wrote: > > Hi, > > I was wondering if someone knowledgeable in IP/CIDR and bitstrings could give me some advice on how to proceed? > > I'm storing network address in an ets table, for example: > 77.242.16.0/20 > 31.24.168.0/21 > > And then I want to check if an IP (31.24.171.44) belongs to any network address in the ets table (the table can get pretty large). > The crux of the matter is that I am not sure which of the following routes to pursue because I only get the IP but CIDR is stored in ets with each network: > > 1. Should I store the network/CIDR in ets as tuples: > {{77, 42, 6, 0}, 20} > {{31, 24, 168, 0}, 21} > and search for the IP as a tuple ({31, 24, 171, 44}) byte by byte? > > 2. Should I store the Network as a bitstring: > <<77,242,1:4>> > <<31,24,21:5>> > And then search for the IP as a bitstring match? > > 3. Should I store the network/CIDR as binary: > <<"77.242.16.0/20">> > <<"31.24.168.0/21">> > > I think the best solution would be the #2 but this one, just like the others I don't know the CIDR ahead of time to be able to trim the CIDR off the IP and then do a simple key based match from ets. > > Any thoughts? > > Thank you > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jose.valim@REDACTED Thu Nov 19 17:48:56 2015 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Fri, 20 Nov 2015 00:48:56 +0800 Subject: [erlang-questions] Mocks and explicit contracts In-Reply-To: <20151119155305.GB40671@fhebert-ltm1> References: <20151119155305.GB40671@fhebert-ltm1> Message-ID: > > First of all, Jos?'s approach also mocks globally in the Erlang context. > It might be different using local modules, but if you enter Erlang, then > it's automatically global. > The article uses a specific definition of the word "mock". So I agree with you that it is still a global mock but we are not mocking the system per the post definition (i.e. changing the implementation during tests). It is a nitpick, sorry, I just wanted to be precise. > Would you consider, for that case, using instead: > > -ifndef(twitter_api). > -define(twitter_api, default_twitter_api). > -endif. > > show(Username) -> > ?twitter_api:get_username(Username). > > Through build tools, you could define (here I'm using rebar3): > > {profiles, [ > [{test, [{erl_opts, [{d, twitter_api, test_twitter_api}]}]}] > ]}. > This is a great solution. I will change the article to use a compile time version of the Elixir implementation as well. Although I would personally not worry about the cost behind accessing an ETS table, specially for this example where we are doing an HTTP request and parsing JSON response for every call, using something akin to a macro is just a better "default". > Q1: Am I going to need multiple modules there because it's a value I want to be configurable by users of a library at runtime? The article was only about handling such cases in your application (no sharing). The thing is, if you want it to be configurable by users, then as you said the application environment is not enough. The good news is that once you provide your configuration mechanism, you have solved the testing problem as well. > Q2: Am I needing multiple modules to test alternating configurations? Yup, I have replied that in another e-mail. It works to some extent as you can give different responses based on the argument. It depends on the system. There are simpler (like passing a closure or an argument) and more complex solutions as well, YMMV. > I personally prefer to pay the cost of slightly longer test runs personally, than having my users pay the cost themselves at run time. That's the only bit I actually disagree. It is not about the performance cost of the test but it is the conceptual cost of mocks. Specially in complex codebases where mocking often hides the fact the developers simply did not define a good contract between their dependencies. That was the main point of the article, anyway. :) Thanks Fred! -------------- next part -------------- An HTML attachment was scrubbed... URL: From axling@REDACTED Thu Nov 19 22:33:06 2015 From: axling@REDACTED (Erik Axling) Date: Thu, 19 Nov 2015 22:33:06 +0100 Subject: [erlang-questions] fannerl, erlang bindings to FANN, the Fast Artificial Neural Network library In-Reply-To: References: Message-ID: Looks like there is a problem encoding large maps (>32 pairs) from a port driver in Erlang/OTP version >= 18.0. Created a bug on Erlang/OTP http://bugs.erlang.org/browse/ERL-47 to clear out if my code is doing something wrong. On Thu, Nov 19, 2015 at 5:19 PM, Erik Axling wrote: > Hello! > > I would like to promote an application that I have made called fannerl. It > is erlang bindings to the FANN, Fast Artificial Neural Networks, library > written by Steffen Nissen. To see more about FANN please check out: > http://leenissen.dk/fann/wp/ > > Fannerl is a port driver that interfaces the libfanndouble version of > FANN. A lot of the functionality is interfaced, not all though, please see > the Github repo for more information. > > Github Link: > https://github.com/axling/fannerl > Docs: > http://axling.github.io/fannerl/ > Releases: > https://github.com/axling/fannerl/releases > > Currently the only release is v1.0.0-alpha. > > The github repo contains examples of how to use fannerl and you can also > read the docs there. > > Consider this in alpha mode as I have only tested this on a Mac OS X > machine as of yet. I would much appreciate your help in testing this on > other platforms. Please report any problems either directly to me, or even > better use the issue tracker in github so more people can see the current > issues. > > Note, Erlang/OTP version must be at least 17.0. FANN version must be at > least 2.2.0. > > Best Regards > Erik Axling > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From snar@REDACTED Thu Nov 19 18:05:10 2015 From: snar@REDACTED (Alexandre Snarskii) Date: Thu, 19 Nov 2015 20:05:10 +0300 Subject: [erlang-questions] IP search optimization In-Reply-To: References: Message-ID: <20151119170510.GA72389@staff.retn.net> On Thu, Nov 19, 2015 at 05:17:21PM +0100, Sid Muller wrote: > Hi, > > I was wondering if someone knowledgeable in IP/CIDR and bitstrings > could give me some advice on how to proceed? > > I'm storing network address in an ets table, for example: > 77.242.16.0/20 > 31.24.168.0/21 > > And then I want to check if an IP (31.24.171.44) belongs to any > network address in the ets table (the table can get pretty large). Looks more like a task for radix trees... The best you can achieve with ets is: on insert: make sure that inserted networks are normalized (do not have any bits set after prefix length). on lookup: 1. start with masklen of 32 2. normalize address to current masklen 3. lookup for address/masklen record, return if found 4. if current masklen is zero - return 'not_found' 5. decrease masklen and repeat from step two. > The crux of the matter is that I am not sure which of the following > routes to pursue because I only get the IP but CIDR is stored in > ets with each network: > > 1. Should I store the network/CIDR in ets as tuples: > {{77, 42, 6, 0}, 20} > {{31, 24, 168, 0}, 21} > and search for the IP as a tuple ({31, 24, 171, 44}) byte by byte? > > 2. Should I store the Network as a bitstring: > <<77,242,1:4>> > <<31,24,21:5>> > And then search for the IP as a bitstring match? > > 3. Should I store the network/CIDR as binary: > <<"77.242.16.0/20">> > <<"31.24.168.0/21">> > > I think the best solution would be the #2 but this one, just like > the others I don't know the CIDR ahead of time to be able to trim > the CIDR off the IP and then do a simple key based match from ets. > > Any thoughts? > > Thank you > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From lheadley@REDACTED Thu Nov 19 21:28:25 2015 From: lheadley@REDACTED (Lyn Headley) Date: Thu, 19 Nov 2015 12:28:25 -0800 Subject: [erlang-questions] Testing a websocket app Message-ID: I'm building an app with yaws, epgsql, poolboy, and websockets. I am looking for advice, recommendations, and/or war stories about about how to set up some tests on this stack. I'd like my test to establish a connection with a web client, send some messages over web sockets, and verify that the responses are correct. Is this something I should use common_test for? What should I/What are you using as a web client? ibrowse? gun? something else? Lyn Headley From dodul@REDACTED Fri Nov 20 00:45:36 2015 From: dodul@REDACTED (Kamrul Khan) Date: Fri, 20 Nov 2015 05:45:36 +0600 Subject: [erlang-questions] Supervising UDP listener Message-ID: I am trying to make a UDP listener that will be supervised by a supervisor module (Just for learning purpose). below id my Supervisor code: -module(pdmanager_sup).-behaviour(supervisor). -export([start_link/1]).-export([init/1]). start_link(Port) ->supervisor:start_link({local,?MODULE}, ?MODULE, Port). init(Port) -> {ok, {{one_for_one, 5, 60}, [{listener, {pdmanager, start_link, Port}, permanent, 1000, worker, [pdmanager]} ]}}. And my udp listener (pdmanager) looks like the below: -module(pdmanager).-behaviour(gen_server). -export([start_link/1]).-export([init/1]).-export([ udplistener/1, handleudp/1]). -record(state, {socket}). start_link(Port) -> gen_server:start_link({local, pdmanager}, pdmanager, Port, []). init(Port) -> io:format("UDP Server starting ~n",[]), {ok, Socket} = gen_udp:open(Port, [binary, {active, false}]), spawn_link(pdmanager,udplistener,[Socket]), {ok, #state{socket=Socket}}. udplistener(Socket) -> {ok,Packet} = gen_udp:recv(Socket,0), {_,_,Msg} = Packet, if Msg /= "stop" -> io:format("Stoping UDP listener ~n", []); true -> spawn(pdmanager,handleudp,[Packet]), udplistener(Socket) end. handleudp(Packet) -> {_,_, Msg} = Packet, io:format("I have got message : ~s ~n",[Msg]), {handeling, Packet}. Now: after compiling I start my program with : pdmanager:start_link(5678). It gets started and I can see the "UDP server starting" message in my screen. Next, when I send a message to UDP 5678 port I see the message "stoping UDP listener" as expected. However, it doesnt restart my server. I called udplistener/1 function from init/1 using spawn_link. I expected spawn_link will link the new process (udp listener) with the calling process (the worker process in this case) and if udplistener goes down the worker will go down aswell and that will cause the supervisor to restart the worker. However, I didnt work that way. How should I do it ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Fri Nov 20 06:56:09 2015 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 19 Nov 2015 21:56:09 -0800 Subject: [erlang-questions] Supervising UDP listener In-Reply-To: References: Message-ID: > On 19 Nov 2015, at 15:45, Kamrul Khan wrote: > > Now: after compiling I start my program with : > > pdmanager:start_link(5678). > Not pdmanager_sup:start_link(5678)? BR, Ulf W -------------- next part -------------- An HTML attachment was scrubbed... URL: From dodul@REDACTED Fri Nov 20 08:33:13 2015 From: dodul@REDACTED (Kamrul Khan) Date: Fri, 20 Nov 2015 13:33:13 +0600 Subject: [erlang-questions] Supervising UDP listener In-Reply-To: References: , Message-ID: Typo. Sorry about that. should be pdmanager_sup:start_link(5678). Subject: Re: [erlang-questions] Supervising UDP listener From: ulf@REDACTED Date: Thu, 19 Nov 2015 21:56:09 -0800 CC: erlang-questions@REDACTED To: dodul@REDACTED On 19 Nov 2015, at 15:45, Kamrul Khan wrote:Now: after compiling I start my program with : pdmanager:start_link(5678). Not pdmanager_sup:start_link(5678)? BR,Ulf W -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Fri Nov 20 09:44:37 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 20 Nov 2015 09:44:37 +0100 Subject: [erlang-questions] Testing a websocket app In-Reply-To: References: Message-ID: On Thu, Nov 19, 2015 at 9:28 PM, Lyn Headley wrote: > Is this something I should use common_test for? What should I/What are > you using as a web client? ibrowse? gun? something else? > My initial thought was gun + common test for this. You may decide if you want the common_test system to run your application, or if it is better to start the app and then let common_test run the gun-parts only, perhaps connected via distribution to the real node. If there is a lot of setup necessary, replicating the configuration in CT is going to take a while. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Fri Nov 20 10:16:27 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Fri, 20 Nov 2015 10:16:27 +0100 Subject: [erlang-questions] Testing a websocket app In-Reply-To: References: Message-ID: I would use detest. Of course I'm biased because I wrote it: https://github.com/biokoda/detest It is an easy way to spawn up one or multiple erlang nodes, run your tests then shut everything down. Sergej On Thu, Nov 19, 2015 at 9:28 PM, Lyn Headley wrote: > I'm building an app with yaws, epgsql, poolboy, and websockets. I am > looking for advice, recommendations, and/or war stories about about > how to set up some tests on this stack. > > I'd like my test to establish a connection with a web client, send > some messages over web sockets, and verify that the responses are > correct. > > Is this something I should use common_test for? What should I/What are > you using as a web client? ibrowse? gun? something else? > > > Lyn Headley > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From viktor@REDACTED Fri Nov 20 11:31:37 2015 From: viktor@REDACTED (=?UTF-8?Q?Viktor_S=c3=b6derqvist?=) Date: Fri, 20 Nov 2015 11:31:37 +0100 Subject: [erlang-questions] How to use crypto:hmac_init() and friends? In-Reply-To: <564DE3FA.9010700@ericsson.com> References: <564DD09C.2040704@xs4all.nl> <564DDA42.7040707@ericsson.com> <564DE3FA.9010700@ericsson.com> Message-ID: <564EF689.80104@zuiderkwast.se> On 2015-11-19 16.00, Sverker Eriksson wrote: > Introducing a new data type in the language is quite a big operation > though. > > These empty magic binaries was sort of a last minute solution > to give NIF designers some way to return "safe pointers" to Erlang. Cool to learn that terms can have emulator state attached. I think the magic could at least be documented. The documentation for http://www.erlang.org/doc/man/crypto.html#hmac_init-2 just says the context is a binary and doesn't mention anything about the emulator state. That would be nice. I certainly would be confused if I ran into this empty binary. -- Viktor S?derqvist -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From biiiipy@REDACTED Fri Nov 20 10:34:38 2015 From: biiiipy@REDACTED (Biiiipy) Date: Fri, 20 Nov 2015 11:34:38 +0200 Subject: [erlang-questions] Support for ECDSA + SHA256 certificates? Message-ID: Hello! I'm using RabbitMQ and I'm having problems with using ECDSA + SHA256 certificates. Is it not supported? Working: ECDSA+SHA1; RSA+SHA1; RSA+SHA256 *Not Working:* ECDSA+SHA256 Running rabbitmq 3.5.6, erlang r18.1, windows 7/linux mint. Certs generated with openssl 1.0.1f and 1.0.2d. Crash Log: ** Reason for termination = ** {function_clause, [{public_key,sign, [{digest, <<57,64,231,69,138,82,239,218,169,237,223,217,16,61,160,220, 38,83,122,74,96,74,92,46,235,162,222,157,233,111,207,245, 237,166,225,110,42,139,85,177,43,201,68,255,30,213,34,130, 253,247,193,2,46,47,152,212,60,82,211,243,9,70,247,136>>}, sha512, {'PrivateKeyInfo',v1, {'PrivateKeyInfo_privateKeyAlgorithm', {1,2,840,10045,2,1}, {asn1_OPENTYPE,<<6,5,43,129,4,0,34>>}}, <<48,129,155,2,1,1,4,48,237,83,11,120,206,206,60,186,204,57, 67,34,207,48,160,53,106,128,99,184,234,248,246,226,38,18, 224,179,13,181,132,114,19,60,30,120,156,126,72,62,95,230, 102,28,152,71,84,22,161,100,3,98,0,4,240,84,45,30,54,167,65, 215,99,151,193,226,179,244,90,187,157,21,201,191,106,28,196, 5,69,246,127,70,247,120,180,66,179,61,88,156,98,238,168,12, 110,114,192,137,183,45,106,177,200,173,116,41,191,190,203, 108,166,91,239,129,27,203,165,160,126,165,9,156,158,145,81, 28,57,88,38,236,146,11,76,38,61,178,65,245,62,52,7,20,26, 123,198,154,162,221,231,2>>, asn1_NOVALUE}], [{file,"public_key.erl"},{line,441}]}, {ssl_handshake,enc_server_key_exchange,6, [{file,"ssl_handshake.erl"},{line,846}]}, {ssl_connection,key_exchange,2, [{file,"ssl_connection.erl"},{line,1245}]}, {ssl_connection,server_certify_and_key_exchange,2, [{file,"ssl_connection.erl"},{line,1142}]}, {ssl_connection,new_server_hello,3, [{file,"ssl_connection.erl"},{line,1015}]}, {tls_connection,next_state,4,[{file,"tls_connection.erl"},{line,466}]}, {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,518}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} =ERROR REPORT==== 16-Nov-2015::14:07:40 === application: mochiweb "Accept failed error" "{'EXIT',\n {{function_clause,\n [{public_key,sign,\n [{digest,\n <<57,64,231,69,138,82,239,218,169,237,223,217,16,61,160,\n 220,38,83,122,74,96,74,92,46,235,162,222,157,233,111,207,\n 245,237,166,225,110,42,139,85,177,43,201,68,255,30,213,\n 34,130,253,247,193,2,46,47,152,212,60,82,211,243,9,70,\n 247,136>>},\n sha512,\n {'PrivateKeyInfo',v1,\n {'PrivateKeyInfo_privateKeyAlgorithm',\n {1,2,840,10045,2,1},\n {asn1_OPENTYPE,<<6,5,43,129,4,0,34>>}},\n <<48,129,155,2,1,1,4,48,237,83,11,120,206,206,60,186,204,57,\n 67,34,207,48,160,53,106,128,99,184,234,248,246,226,38,18,\n 224,179,13,181,132,114,19,60,30,120,156,126,72,62,95,230,\n 102,28,152,71,84,22,161,100,3,98,0,4,240,84,45,30,54,167,\n 65,215,99,151,193,226,179,244,90,187,157,21,201,191,106,\n 28,196,5,69,246,127,70,247,120,180,66,179,61,88,156,98,\n 238,168,12,110,114,192,137,183,45,106,177,200,173,116,41,\n 191,190,203,108,166,91,239,129,27,203,165,160,126,165,9,\n 156,158,145,81,28,57,88,38,236,146,11,76,38,61,178,65,245,\n 62,52,7,20,26,123,198,154,162,221,231,2>>,\n asn1_NOVALUE}],\n [{file,\"public_key.erl\"},{line,441}]},\n {ssl_handshake,enc_server_key_exchange,6,\n [{file,\"ssl_handshake.erl\"},{line,846}]},\n {ssl_connection,key_exchange,2,\n [{file,\"ssl_connection.erl\"},{line,1245}]},\n {ssl_connection,server_certify_and_key_exchange,2,\n [{file,\"ssl_connection.erl\"},{line,1142}]},\n {ssl_connection,new_server_hello,3,\n [{file,\"ssl_connection.erl\"},{line,1015}]},\n {tls_connection,next_state,4,\n [{file,\"tls_connection.erl\"},{line,466}]},\n {gen_fsm,handle_msg,7,[{file,\"gen_fsm.erl\"},{line,518}]},\n {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]},\n {gen_fsm,sync_send_all_state_event,\n [<0.533.0>,{start,infinity},infinity]}}}" -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Fri Nov 20 12:43:25 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 20 Nov 2015 12:43:25 +0100 Subject: [erlang-questions] How to use crypto:hmac_init() and friends? In-Reply-To: <564EF689.80104@zuiderkwast.se> References: <564DD09C.2040704@xs4all.nl> <564DDA42.7040707@ericsson.com> <564DE3FA.9010700@ericsson.com> <564EF689.80104@zuiderkwast.se> Message-ID: On Fri, Nov 20, 2015 at 11:31 AM, Viktor S?derqvist wrote: > I think the magic could at least be documented. The documentation for > http://www.erlang.org/doc/man/crypto.html#hmac_init-2 just says the > context is a binary and doesn't mention anything about the emulator > state. That would be nice. I certainly would be confused if I ran into > this empty binary. > It feels like a hack to me. Having an explicit opaque type for these kinds of things would be really nice. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dch@REDACTED Fri Nov 20 14:20:04 2015 From: dch@REDACTED (Dave Cottlehuber) Date: Fri, 20 Nov 2015 14:20:04 +0100 Subject: [erlang-questions] race condition on BEAM startup before DNS queries function Message-ID: <1448025604.2099447.445279537.4BD90311@webmail.messagingengine.com> It seems there is some module or application that is required as a dependency for e.g. DNS queries to work reliably, when the BEAM starts up [1]. If the delay is set to 5 seconds, this lookup works reliably, if it is 0, then it fails reliably: erl -eval 'timer:sleep(5000), io:format("~p~n", [inet_res:lookup("erlang.org", in, mx)]).' I'm stumped as to what this might be, and early attempts at debugging created a bit too much logs for my liking ... ideas? [1]: http://stackoverflow.com/questions/33811899/elixir-errors-using-inet-res-library A+ Dave ? Dave Cottlehuber Skunkwerks, GmbH From jim.rosenblum@REDACTED Fri Nov 20 15:03:43 2015 From: jim.rosenblum@REDACTED (jim rosenblum) Date: Fri, 20 Nov 2015 09:03:43 -0500 Subject: [erlang-questions] guard clause oddity, maybe bug (in my brain)? Message-ID: I have a function: walk ([I|Is], [Elt|_]=Ts) when (not is_tuple(Elt)) andalso ?IS_INDEX(I) -> Element = index_to_element(I,Ts), case ... snip ... end. This function clause gets executed even though Elt is a tuple. I have a break point within this function clause. I am looking at the debugger at the break-point within the clause, and I can plainly see that Elt IS a tuple: within the debugger I can evaluate is_tuple(Elt) and it returns true, I can evaluate (not is_tuple(Elt)) and correctly get FALSE. So.... how can I be in this clause when the guard evaluates to false? Thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Fri Nov 20 15:25:43 2015 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 20 Nov 2015 15:25:43 +0100 Subject: [erlang-questions] guard clause oddity, maybe bug (in my brain)? In-Reply-To: References: Message-ID: <564F2D67.2030307@ericsson.com> Greetings, I can not duplicate your problem. In case I have misunderstood, here is my code (-define is not my strong suite, so it has been removed). t() -> walk( [asd, qwe], [{123, 456}] ). walk ([I|Is], [Elt|_]=Ts) when (not is_tuple(Elt)) -> {[I|Is], Ts}; walk(_, _) -> the_other. This is what happens: 4> c(t). {ok,t} 5> t:t(). the_other bengt On 11/20/2015 03:03 PM, jim rosenblum wrote: > I have a function: > > walk ([I|Is], [Elt|_]=Ts) when (not is_tuple(Elt)) andalso ?IS_INDEX(I) -> > Element = index_to_element(I,Ts), > case > ... snip ... > end. > > This function clause gets executed even though Elt is a tuple. > > I have a break point within this function clause. I am looking at the > debugger at the break-point within the clause, and I can plainly see > that Elt > IS a tuple: within the debugger I can evaluate is_tuple(Elt) and it > returns true, > I can evaluate (not is_tuple(Elt)) and correctly get FALSE. > > So.... how can I be in this clause when the guard evaluates to false? > > Thoughts? > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Fri Nov 20 16:03:01 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 20 Nov 2015 17:03:01 +0200 Subject: [erlang-questions] guard clause oddity, maybe bug (in my brain)? In-Reply-To: References: Message-ID: <08B13D4A-EFEE-412C-8BE2-180BE428F009@gmail.com> Hello, How does you macro IS_INDEX looks like? e.g. 5> A = {1}. {1} 6> (not is_tuple(A)) andalso is_number(A) orelse A > 1. true but 8> (not is_tuple(A)) andalso (is_number(A) orelse A > 1). false Best Regards, Dmitry > On Nov 20, 2015, at 4:03 PM, jim rosenblum wrote: > > I have a function: > > walk ([I|Is], [Elt|_]=Ts) when (not is_tuple(Elt)) andalso ?IS_INDEX(I) -> > Element = index_to_element(I,Ts), > case > ... snip ... > end. > > This function clause gets executed even though Elt is a tuple. > > I have a break point within this function clause. I am looking at the > debugger at the break-point within the clause, and I can plainly see that Elt > IS a tuple: within the debugger I can evaluate is_tuple(Elt) and it returns true, > I can evaluate (not is_tuple(Elt)) and correctly get FALSE. > > So.... how can I be in this clause when the guard evaluates to false? > > Thoughts? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sid5@REDACTED Fri Nov 20 17:42:13 2015 From: sid5@REDACTED (Sid Muller) Date: Fri, 20 Nov 2015 17:42:13 +0100 Subject: [erlang-questions] bitstring ets:select Message-ID: Is there a way to do a select on a bitstring based on a value in ets table? The X:'$2'/bitstring below results in an illegal bit size, which makes sense because '$2' doesn't exits yet. How do I dynamically create that X value based on external B value and '$2' that will be selected from ets table? Is that even possible? tst()-> table = ets:new(table, [set, named_table]), A = <<2#000111110001100010101:21>>, true = ets:insert(table, {A, 21}), B = <<31,24,171,44>>, Ret = ets:select(table, [{{'$1','$2'},[{'==','$1', fun()-><> = B, X end}],[['$1', '$2']]}]), ets:delete(table). From sid5@REDACTED Fri Nov 20 18:01:50 2015 From: sid5@REDACTED (Sid Muller) Date: Fri, 20 Nov 2015 18:01:50 +0100 Subject: [erlang-questions] IP search optimization In-Reply-To: <20151119170510.GA72389@staff.retn.net> References: , <20151119170510.GA72389@staff.retn.net> Message-ID: > Sent: Thursday, November 19, 2015 at 9:05 AM > From: "Alexandre Snarskii" > To: "Sid Muller" > Cc: erlang-questions > Subject: Re: [erlang-questions] IP search optimization > > And then I want to check if an IP (31.24.171.44) belongs to any > > network address in the ets table (the table can get pretty large). > > Looks more like a task for radix trees... Hmm, good point although with ets if my owner process gets killed I inherit the ets and no data is lost. Not sure how that cold be done with a radix tree. > The best you can achieve with ets is: > on insert: make sure that inserted networks are normalized (do not have > any bits set after prefix length). > on lookup: > 1. start with masklen of 32 > 2. normalize address to current masklen > 3. lookup for address/masklen record, return if found > 4. if current masklen is zero - return 'not_found' > 5. decrease masklen and repeat from step two. There are a few faster methods: 1. Have a pre-scanned list of /CIDR [21, 20...], there are usually about 10. But this still traverses the table 10 times in worst case. 2. Traverse the table with ets:lookup(ets:next(), this has the advantage over #1 as it's much faster in most cases. 3. ets:select(MatchSpec), this is probably the fastest solution because it should traverse the table only once. But right now I'm having a bit of a problem with my select statement... see other email [bitstring select] From dodul@REDACTED Fri Nov 20 18:38:25 2015 From: dodul@REDACTED (Kamrul Khan) Date: Fri, 20 Nov 2015 23:38:25 +0600 Subject: [erlang-questions] Supervising UDP listener In-Reply-To: References: , , , Message-ID: I didnt get any response for my question. I am posting it again ( with the typo fix) I am trying to make a UDP listener that will be supervised by a supervisor module (Just for learning purpose). below id my Supervisor code:-module(pdmanager_sup).-behaviour(supervisor). -export([start_link/1]).-export([init/1]). start_link(Port) ->supervisor:start_link({local,?MODULE}, ?MODULE, Port). init(Port) -> {ok, {{one_for_one, 5, 60}, [{listener, {pdmanager, start_link, Port}, permanent, 1000, worker, [pdmanager]} ]}}.And my udp listener (pdmanager) looks like the below:-module(pdmanager).-behaviour(gen_server).-export([start_link/1]).-export([init/1]).-export([ udplistener/1, handleudp/1]).-record(state, {socket}).start_link(Port) -> gen_server:start_link({local, pdmanager}, pdmanager, Port, []).init(Port) -> io:format("UDP Server starting ~n",[]), {ok, Socket} = gen_udp:open(Port, [binary, {active, false}]), spawn_link(pdmanager,udplistener,[Socket]), {ok, #state{socket=Socket}}.udplistener(Socket) -> {ok,Packet} = gen_udp:recv(Socket,0), {_,_,Msg} = Packet, if Msg /= "stop" -> io:format("Stoping UDP listener ~n", []); true -> spawn(pdmanager,handleudp,[Packet]), udplistener(Socket) end. handleudp(Packet) -> {_,_, Msg} = Packet, io:format("I have got message : ~s ~n",[Msg]), {handeling, Packet}.Now: after compiling I start my program with :pdmanager_sup:start_link(5678).It gets started and I can see the "UDP server starting" message in my screen. Next, when I send a message to UDP 5678 port I see the message "stoping UDP listener" as expected. However, it doesnt restart my server. I called udplistener/1 function from init/1 using spawn_link. I expected spawn_link will link the new process (udp listener) with the calling process (the worker process in this case) and if udplistener goes down the worker will go down aswell and that will cause the supervisor to restart the worker. However, I didnt work that way. How should I do it ? From: dodul@REDACTED To: ulf@REDACTED Date: Fri, 20 Nov 2015 13:33:13 +0600 CC: erlang-questions@REDACTED Subject: Re: [erlang-questions] Supervising UDP listener Typo. Sorry about that. should be pdmanager_sup:start_link(5678). Subject: Re: [erlang-questions] Supervising UDP listener From: ulf@REDACTED Date: Thu, 19 Nov 2015 21:56:09 -0800 CC: erlang-questions@REDACTED To: dodul@REDACTED On 19 Nov 2015, at 15:45, Kamrul Khan wrote:Now: after compiling I start my program with : pdmanager:start_link(5678). Not pdmanager_sup:start_link(5678)? BR,Ulf W _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve@REDACTED Fri Nov 20 19:53:11 2015 From: steve@REDACTED (Steve Strong) Date: Fri, 20 Nov 2015 18:53:11 +0000 Subject: [erlang-questions] Supervising UDP listener In-Reply-To: References: Message-ID: You create udp_listener with spawn_link, which would kill the parent *if* udp_listener terminates abnormally. In your case, udp_listener is exiting cleanly, so it won't cause the parent to die. Cheers. Steve -- Steve Strong Sent with Airmail On 20 Nov 2015, 17:38 +0000, Kamrul Khan, wrote: > I didnt get any response for my question. I am posting it again ( with the typo fix) > > I am trying to make a UDP listener that will be supervised by a supervisor module (Just for learning purpose).below id my Supervisor code: > > -module(pdmanager_sup). > -behaviour(supervisor). > > -export([start_link/1]). > -export([init/1]). > > start_link(Port) -> > supervisor:start_link({local,?MODULE}, ?MODULE, Port). > > init(Port) -> > {ok, {{one_for_one, 5, 60}, > [{listener, > {pdmanager, start_link, Port}, > permanent, 1000, worker, [pdmanager]} > ]}}. > > > And my udp listener (pdmanager) looks like the below: > > -module(pdmanager). > -behaviour(gen_server). > > -export([start_link/1]). > -export([init/1]). > -export([ > udplistener/1, > handleudp/1 > ]). > > -record(state, {socket}). > > start_link(Port) -> > gen_server:start_link({local, pdmanager}, pdmanager, Port, []). > > init(Port) -> > io:format("UDP Server starting ~n",[]), > {ok, Socket} = gen_udp:open(Port, [binary, {active, false}]), > spawn_link(pdmanager,udplistener,[Socket]), > {ok, #state{socket=Socket}}. > > udplistener(Socket) -> > {ok,Packet} = gen_udp:recv(Socket,0), > {_,_,Msg} = Packet, > if > Msg /= "stop" -> > io:format("Stoping UDP listener ~n", []); > true -> > spawn(pdmanager,handleudp,[Packet]), > udplistener(Socket) > end. > > handleudp(Packet) -> > {_,_, Msg} = Packet, > io:format("I have got message : ~s ~n",[Msg]), > {handeling, Packet}. > > > > Now: after compiling I start my program with : > > pdmanager_sup:start_link(5678). > > It gets started and I can see the "UDP server starting" message in my screen. Next, when I send a message to UDP 5678 port I see the message "stoping UDP listener" as expected.However, it doesnt restart my server.I called udplistener/1 function from init/1 using spawn_link. I expected spawn_link will link the new process (udp listener) with the calling process (the worker process in this case) and if udplistener goes down the worker will go down aswell and that will cause the supervisor to restart the worker. However, I didnt work that way. > > How should I do it ? > From: dodul@REDACTED > To: ulf@REDACTED > Date: Fri, 20 Nov 2015 13:33:13 +0600 > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Supervising UDP listener > > Typo. Sorry about that. should be > pdmanager_sup:start_link(5678). > > Subject: Re: [erlang-questions] Supervising UDP listener > From: ulf@REDACTED > Date: Thu, 19 Nov 2015 21:56:09 -0800 > CC: erlang-questions@REDACTED > To: dodul@REDACTED > > > > On 19 Nov 2015, at 15:45, Kamrul Khanwrote: > > Now: after compiling I start my program with : > > > > pdmanager:start_link(5678). > > Not pdmanager_sup:start_link(5678)? > > BR, > Ulf W > > > > > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dodul@REDACTED Fri Nov 20 20:17:11 2015 From: dodul@REDACTED (Kamrul Khan) Date: Sat, 21 Nov 2015 01:17:11 +0600 Subject: [erlang-questions] Supervising UDP listener In-Reply-To: References: , , , , Message-ID: Hello Steve, Thanks for your response. I want to check if the supervisor is doing its job. Whats the best way for me to simulate an abnormal termination. CheersKamrul Date: Fri, 20 Nov 2015 18:53:11 +0000 From: steve@REDACTED To: ulf@REDACTED; dodul@REDACTED CC: erlang-questions@REDACTED Subject: Re: [erlang-questions] Supervising UDP listener You create udp_listener with spawn_link, which would kill the parent *if* udp_listener terminates abnormally. In your case, udp_listener is exiting cleanly, so it won't cause the parent to die. Cheers. Steve -- Steve Strong Sent with Airmail On 20 Nov 2015, 17:38 +0000, Kamrul Khan , wrote: I didnt get any response for my question. I am posting it again ( with the typo fix) I am trying to make a UDP listener that will be supervised by a supervisor module (Just for learning purpose). below id my Supervisor code: -module(pdmanager_sup). -behaviour(supervisor). -export([start_link/1]). -export([init/1]). start_link(Port) -> supervisor:start_link({local,?MODULE}, ?MODULE, Port). init(Port) -> {ok, {{one_for_one, 5, 60}, [{listener, {pdmanager, start_link, Port}, permanent, 1000, worker, [pdmanager]} ]}}. And my udp listener (pdmanager) looks like the below: -module(pdmanager). -behaviour(gen_server). -export([start_link/1]). -export([init/1]). -export([ udplistener/1, handleudp/1 ]). -record(state, {socket}). start_link(Port) -> gen_server:start_link({local, pdmanager}, pdmanager, Port, []). init(Port) -> io:format("UDP Server starting ~n",[]), {ok, Socket} = gen_udp:open(Port, [binary, {active, false}]), spawn_link(pdmanager,udplistener,[Socket]), {ok, #state{socket=Socket}}. udplistener(Socket) -> {ok,Packet} = gen_udp:recv(Socket,0), {_,_,Msg} = Packet, if Msg /= "stop" -> io:format("Stoping UDP listener ~n", []); true -> spawn(pdmanager,handleudp,[Packet]), udplistener(Socket) end. handleudp(Packet) -> {_,_, Msg} = Packet, io:format("I have got message : ~s ~n",[Msg]), {handeling, Packet}. Now: after compiling I start my program with : pdmanager_sup:start_link(5678). It gets started and I can see the "UDP server starting" message in my screen. Next, when I send a message to UDP 5678 port I see the message "stoping UDP listener" as expected. However, it doesnt restart my server. I called udplistener/1 function from init/1 using spawn_link. I expected spawn_link will link the new process (udp listener) with the calling process (the worker process in this case) and if udplistener goes down the worker will go down aswell and that will cause the supervisor to restart the worker. However, I didnt work that way. How should I do it ? From: dodul@REDACTED To: ulf@REDACTED Date: Fri, 20 Nov 2015 13:33:13 +0600 CC: erlang-questions@REDACTED Subject: Re: [erlang-questions] Supervising UDP listener Typo. Sorry about that. should be pdmanager_sup:start_link(5678). Subject: Re: [erlang-questions] Supervising UDP listener From: ulf@REDACTED Date: Thu, 19 Nov 2015 21:56:09 -0800 CC: erlang-questions@REDACTED To: dodul@REDACTED On 19 Nov 2015, at 15:45, Kamrul Khan wrote: Now: after compiling I start my program with : pdmanager:start_link(5678). Not pdmanager_sup:start_link(5678)? BR, Ulf W _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve@REDACTED Fri Nov 20 20:24:57 2015 From: steve@REDACTED (Steve Strong) Date: Fri, 20 Nov 2015 19:24:57 +0000 Subject: [erlang-questions] Supervising UDP listener In-Reply-To: References: Message-ID: After your io:format, do something like exit(error). That should cause the parent process to exit and get restarted by the supervisor.http://www.erlang.org/doc/reference_manual/processes.html(http://www.erlang.org/doc/reference_manual/processes.html#errors)has the info on this. Cheers. Steve -- Steve Strong Sent with Airmail On 20 Nov 2015, 19:17 +0000, Kamrul Khan, wrote: > Hello Steve, > > Thanks for your response. I want to check if the supervisor is doing its job. Whats the best way for me to simulate an abnormal termination. > > Cheers > Kamrul > Date: Fri, 20 Nov 2015 18:53:11 +0000 > From: steve@REDACTED > To: ulf@REDACTED; dodul@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Supervising UDP listener > > You create udp_listener with spawn_link, which would kill the parent *if* udp_listener terminates abnormally. In your case, udp_listener is exiting cleanly, so it won't cause the parent to die. > > > Cheers. > > Steve > > -- > Steve Strong > Sent with Airmail > > On 20 Nov 2015, 17:38 +0000, Kamrul Khan, wrote: > > I didnt get any response for my question. I am posting it again ( with the typo fix) > > > > I am trying to make a UDP listener that will be supervised by a supervisor module (Just for learning purpose).below id my Supervisor code: > > > > -module(pdmanager_sup). > > -behaviour(supervisor). > > > > -export([start_link/1]). > > -export([init/1]). > > > > start_link(Port) -> > > supervisor:start_link({local,?MODULE}, ?MODULE, Port). > > > > init(Port) -> > > {ok, {{one_for_one, 5, 60}, > > [{listener, > > {pdmanager, start_link, Port}, > > permanent, 1000, worker, [pdmanager]} > > ]}}. > > > > > > And my udp listener (pdmanager) looks like the below: > > > > -module(pdmanager). > > -behaviour(gen_server). > > > > -export([start_link/1]). > > -export([init/1]). > > -export([ > > udplistener/1, > > handleudp/1 > > ]). > > > > -record(state, {socket}). > > > > start_link(Port) -> > > gen_server:start_link({local, pdmanager}, pdmanager, Port, []). > > > > init(Port) -> > > io:format("UDP Server starting ~n",[]), > > {ok, Socket} = gen_udp:open(Port, [binary, {active, false}]), > > spawn_link(pdmanager,udplistener,[Socket]), > > {ok, #state{socket=Socket}}. > > > > udplistener(Socket) -> > > {ok,Packet} = gen_udp:recv(Socket,0), > > {_,_,Msg} = Packet, > > if > > Msg /= "stop" -> > > io:format("Stoping UDP listener ~n", []); > > true -> > > spawn(pdmanager,handleudp,[Packet]), > > udplistener(Socket) > > end. > > > > handleudp(Packet) -> > > {_,_, Msg} = Packet, > > io:format("I have got message : ~s ~n",[Msg]), > > {handeling, Packet}. > > > > > > > > Now: after compiling I start my program with : > > > > pdmanager_sup:start_link(5678). > > > > It gets started and I can see the "UDP server starting" message in my screen. Next, when I send a message to UDP 5678 port I see the message "stoping UDP listener" as expected.However, it doesnt restart my server.I called udplistener/1 function from init/1 using spawn_link. I expected spawn_link will link the new process (udp listener) with the calling process (the worker process in this case) and if udplistener goes down the worker will go down aswell and that will cause the supervisor to restart the worker. However, I didnt work that way. > > > > How should I do it ? > > From: dodul@REDACTED > > To: ulf@REDACTED > > Date: Fri, 20 Nov 2015 13:33:13 +0600 > > CC: erlang-questions@REDACTED > > Subject: Re: [erlang-questions] Supervising UDP listener > > > > Typo. Sorry about that. should be > > pdmanager_sup:start_link(5678). > > > > Subject: Re: [erlang-questions] Supervising UDP listener > > From: ulf@REDACTED > > Date: Thu, 19 Nov 2015 21:56:09 -0800 > > CC: erlang-questions@REDACTED > > To: dodul@REDACTED > > > > > > > On 19 Nov 2015, at 15:45, Kamrul Khanwrote: > > > Now: after compiling I start my program with : > > > > > > pdmanager:start_link(5678). > > > > Not pdmanager_sup:start_link(5678)? > > > > BR, > > Ulf W > > > > > > > > > > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Fri Nov 20 21:46:29 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Fri, 20 Nov 2015 21:46:29 +0100 Subject: [erlang-questions] guard clause oddity, maybe bug (in my brain)? In-Reply-To: <08B13D4A-EFEE-412C-8BE2-180BE428F009@gmail.com> References: <08B13D4A-EFEE-412C-8BE2-180BE428F009@gmail.com> Message-ID: <2AA4A1FA-0F32-4B6D-94EB-4B5185A2ACD0@gmail.com> Dmitry is on point. 1. Remove the useless braces around not is_tuple... 2. Add braces around ?IS_INDEX... If that fixes it, think about moving the braces inside the macro's definition. > On 20 Nov 2015, at 16:03, Dmitry Kolesnikov wrote: > > Hello, > > How does you macro IS_INDEX looks like? > e.g. > > 5> A = {1}. > {1} > 6> (not is_tuple(A)) andalso is_number(A) orelse A > 1. > true > > but > > 8> (not is_tuple(A)) andalso (is_number(A) orelse A > 1). > false > > > Best Regards, > Dmitry > > >> On Nov 20, 2015, at 4:03 PM, jim rosenblum wrote: >> >> I have a function: >> >> walk ([I|Is], [Elt|_]=Ts) when (not is_tuple(Elt)) andalso ?IS_INDEX(I) -> >> Element = index_to_element(I,Ts), >> case >> ... snip ... >> end. >> >> This function clause gets executed even though Elt is a tuple. >> >> I have a break point within this function clause. I am looking at the >> debugger at the break-point within the clause, and I can plainly see that Elt >> IS a tuple: within the debugger I can evaluate is_tuple(Elt) and it returns true, >> I can evaluate (not is_tuple(Elt)) and correctly get FALSE. >> >> So.... how can I be in this clause when the guard evaluates to false? >> >> Thoughts? >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From felixgallo@REDACTED Fri Nov 20 23:46:57 2015 From: felixgallo@REDACTED (Felix Gallo) Date: Fri, 20 Nov 2015 14:46:57 -0800 Subject: [erlang-questions] Mersenne Twister Message-ID: In 2011 there was discussion on this list around improving the stdlib random number generator, perhaps to join the crowd of languages using Mersenne Twister or a variant. As far as I can tell, that's about as far as anyone got; it appears that random:uniform() is still Wichmann-Hill, which is apparently held in low regard by randomness fans. Are there any extant plans to replace it with a more modern PRNG, such as MT? If they don't exist, what's the OTP team's current thoughts regarding the likelihood of accepting such a patch? F. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Nov 20 23:51:29 2015 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 20 Nov 2015 23:51:29 +0100 Subject: [erlang-questions] bitstring ets:select In-Reply-To: References: Message-ID: On Fri, Nov 20, 2015 at 5:42 PM, Sid Muller wrote: > Is there a way to do a select on a bitstring based on a value in ets table? > The X:'$2'/bitstring below results in an illegal bit size, which makes > sense because '$2' doesn't exits yet. > > How do I dynamically create that X value based on external B value and > '$2' that will be selected from ets table? Is that even possible? > As far as I know, it is not possible. Match specs, as you may have noticed, do not have a lot of support for working with binaries, and even if you substitute your binary to a list, it is not possible to do what you want to do. The language is (afaik) not powerful enough to express that kind of logic. If you know the length of the prefix beforehand, you could do it, but if you don't then it becomes very hard/impossible to do with what is allowed in match specs today. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Fri Nov 20 23:53:03 2015 From: felixgallo@REDACTED (Felix Gallo) Date: Fri, 20 Nov 2015 14:53:03 -0800 Subject: [erlang-questions] Mersenne Twister In-Reply-To: References: Message-ID: Aha, Jesper points out the rand namespace to me, which looks like it provides a bunch more PRNGs as of R18. Thanks Jesper and, apparently, Kenji et al. F. On Fri, Nov 20, 2015 at 2:46 PM, Felix Gallo wrote: > In 2011 there was discussion on this list around improving the stdlib > random number generator, perhaps to join the crowd of languages using > Mersenne Twister or a variant. > > As far as I can tell, that's about as far as anyone got; it appears that > random:uniform() is still Wichmann-Hill, which is apparently held in low > regard by randomness fans. > > Are there any extant plans to replace it with a more modern PRNG, such as > MT? If they don't exist, what's the OTP team's current thoughts regarding > the likelihood of accepting such a patch? > > F. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nem@REDACTED Sat Nov 21 00:52:12 2015 From: nem@REDACTED (Geoff Cant) Date: Fri, 20 Nov 2015 15:52:12 -0800 Subject: [erlang-questions] IP search optimization In-Reply-To: References: Message-ID: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> Hi Sid, are you writing an open source library for this functionality? I need exactly the same thing and was going to sit down to write it. With CIDR blocks, you can always calculate the lower and upper address in a range. > -type ip4() :: <<_:32/bits>>. > -type ip4_mask() :: 0..32. > -type ip4_cidr() :: {ip4(), ip4_mask()}. > -spec to_range(ip4_cidr()) -> {Low::ip4(), High::ip4()}. > to_range({<>, Mask}) > when 0 =< Mask, > Mask =< 32 -> > HostBits = 32 - Mask, > {<>, > <>}. If you pre-convert your CIDR list into a range list, you can then select all the matching cidrs by comparing Low =< IP, IP =< High, and returning the CIDR with the largest mask (if you do longest-prefix wins), or the first in order if you?re doing an ordered CIDR match. Pre-converting CIDR -> Range will let you use ets:select - and might be reasonably efficient in an ordered_set table if you?ve got large numbers of CIDRs. I think if your ets key is {Low, High}, and it?s an ordered_set table, the first result from a matching select will be the one with the longest Mask anyway (as {<<192,168,0,0>>, <<192,168,0,255>>} < {<<192,168,0,0>>, <<192,168,255,255>>} ? i.e. 192.168.0.0/24 < 192.168.0.0/16 when converted to a range tuple). > IP = something, > Matching = ets:select(cidr_table, ets:fun2ms(fun (#record{range={Low, High}}) when Low =< IP, IP =< High) -> object() end)). Anyway - I?m either going to write something like this in the next couple of weeks, or use yours if it?s public and suitable :) Cheers, -Geoff > On 19 Nov, 2015, at 08:17, Sid Muller wrote: > > Hi, > > I was wondering if someone knowledgeable in IP/CIDR and bitstrings could give me some advice on how to proceed? > > I'm storing network address in an ets table, for example: > 77.242.16.0/20 > 31.24.168.0/21 > > And then I want to check if an IP (31.24.171.44) belongs to any network address in the ets table (the table can get pretty large). > The crux of the matter is that I am not sure which of the following routes to pursue because I only get the IP but CIDR is stored in ets with each network: > > 1. Should I store the network/CIDR in ets as tuples: > {{77, 42, 6, 0}, 20} > {{31, 24, 168, 0}, 21} > and search for the IP as a tuple ({31, 24, 171, 44}) byte by byte? > > 2. Should I store the Network as a bitstring: > <<77,242,1:4>> > <<31,24,21:5>> > And then search for the IP as a bitstring match? > > 3. Should I store the network/CIDR as binary: > <<"77.242.16.0/20">> > <<"31.24.168.0/21">> > > I think the best solution would be the #2 but this one, just like the others I don't know the CIDR ahead of time to be able to trim the CIDR off the IP and then do a simple key based match from ets. > > Any thoughts? > > Thank you > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From kennethlakin@REDACTED Sat Nov 21 03:29:18 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Fri, 20 Nov 2015 18:29:18 -0800 Subject: [erlang-questions] race condition on BEAM startup before DNS queries function In-Reply-To: <1448025604.2099447.445279537.4BD90311@webmail.messagingengine.com> References: <1448025604.2099447.445279537.4BD90311@webmail.messagingengine.com> Message-ID: <564FD6FE.7000808@gmail.com> On 11/20/2015 05:20 AM, Dave Cottlehuber wrote: > It seems there is some module or application that is required as a > dependency for e.g. DNS queries to work reliably, when the BEAM starts > up [1]. > > If the delay is set to 5 seconds, this lookup works reliably, if it is > 0, then it fails reliably: It *feels* like the following logic in res_update/5 (in inet_db.erl) is the culprit: res_update(Tag, TagTm, TagInfo, TagSetTm, SetFun) -> case db_get(TagTm) of undefined -> ok; TM -> case times() of Now when Now >= TM + ?RES_FILE_UPDATE_TM -> % _ -> ok end end. RES_FILE_UPDATE_TM is defined as 5. The comment next to it declares that its unit is seconds. inet_db:times/0 is times() -> erlang:convert_time_unit(erlang:monotonic_time() - erlang:system_info(start_time),native,seconds). inet_db:times/0 appears to return the number of seconds the system has been up. Some tracing and prints reveals that the nameservers list passed in to the call to inet_res:do_query is *not* populated until the system has been up and running for ~five seconds (that is, until inet_db:times/0 returns 5). The call immediately prior to the times/0 call is inet_db:db_get(res_resolv_conf_tm) This ends up being a lookup into the inet_db ETS table. I don't know what the value of res_resolv_conf_tm is, but maybe it *might* be possible to set it to -?RES_FILE_UPDATE_TM so that DNS lookups will load the nameservers defined in resolv.conf when the system initially starts up, rather than having to wait for five seconds? But maybe I'm wrong about all of that. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From mjtruog@REDACTED Sat Nov 21 04:40:08 2015 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 20 Nov 2015 19:40:08 -0800 Subject: [erlang-questions] IP search optimization In-Reply-To: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> References: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> Message-ID: <564FE798.5050305@gmail.com> Comment below: On 11/20/2015 03:52 PM, Geoff Cant wrote: > Hi Sid, are you writing an open source library for this functionality? I need exactly the same thing and was going to sit down to write it. > > With CIDR blocks, you can always calculate the lower and upper address in a range. > >> -type ip4() :: <<_:32/bits>>. >> -type ip4_mask() :: 0..32. >> -type ip4_cidr() :: {ip4(), ip4_mask()}. >> -spec to_range(ip4_cidr()) -> {Low::ip4(), High::ip4()}. >> to_range({<>, Mask}) >> when 0 =< Mask, >> Mask =< 32 -> >> HostBits = 32 - Mask, >> {<>, >> <>}. > If you pre-convert your CIDR list into a range list, you can then select all the matching cidrs by comparing Low =< IP, IP =< High, and returning the CIDR with the largest mask (if you do longest-prefix wins), or the first in order if you?re doing an ordered CIDR match. > > Pre-converting CIDR -> Range will let you use ets:select - and might be reasonably efficient in an ordered_set table if you?ve got large numbers of CIDRs. I think if your ets key is {Low, High}, and it?s an ordered_set table, the first result from a matching select will be the one with the longest Mask anyway (as {<<192,168,0,0>>, <<192,168,0,255>>} < {<<192,168,0,0>>, <<192,168,255,255>>} ? i.e. 192.168.0.0/24 < 192.168.0.0/16 when converted to a range tuple). If you used https://github.com/okeuday/trie with an Erlang string (list of integers) it would be faster than the ets usage, though it can consume more memory if it needs to be replicated in many Erlang processes. The basic way would be with a decimal number in each element, so [192,168,0,0] with the range expanded (as described above) to store everything. You could also use the trie:find_match/2 function with the IP address stored as a boolean string with a wildcard character to represent the range, e.g.: lists:sublist(lists:flatten(io_lib:format("~8.2.0B~8.2.0B~8.2.0B~8.2.0B", [192, 168, 0, 0])), 16) ++ "*". % is 192.168.0.0/16 in a format to use with trie:store/3 as the key argument then just use find_match/2 to match the IP patterns (the most specific match would be found) Best Regards, Michael > >> IP = something, >> Matching = ets:select(cidr_table, ets:fun2ms(fun (#record{range={Low, High}}) when Low =< IP, IP =< High) -> object() end)). > > Anyway - I?m either going to write something like this in the next couple of weeks, or use yours if it?s public and suitable :) > > Cheers, > -Geoff > > >> On 19 Nov, 2015, at 08:17, Sid Muller wrote: >> >> Hi, >> >> I was wondering if someone knowledgeable in IP/CIDR and bitstrings could give me some advice on how to proceed? >> >> I'm storing network address in an ets table, for example: >> 77.242.16.0/20 >> 31.24.168.0/21 >> >> And then I want to check if an IP (31.24.171.44) belongs to any network address in the ets table (the table can get pretty large). >> The crux of the matter is that I am not sure which of the following routes to pursue because I only get the IP but CIDR is stored in ets with each network: >> >> 1. Should I store the network/CIDR in ets as tuples: >> {{77, 42, 6, 0}, 20} >> {{31, 24, 168, 0}, 21} >> and search for the IP as a tuple ({31, 24, 171, 44}) byte by byte? >> >> 2. Should I store the Network as a bitstring: >> <<77,242,1:4>> >> <<31,24,21:5>> >> And then search for the IP as a bitstring match? >> >> 3. Should I store the network/CIDR as binary: >> <<"77.242.16.0/20">> >> <<"31.24.168.0/21">> >> >> I think the best solution would be the #2 but this one, just like the others I don't know the CIDR ahead of time to be able to trim the CIDR off the IP and then do a simple key based match from ets. >> >> Any thoughts? >> >> Thank you >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jim.rosenblum@REDACTED Sat Nov 21 05:27:42 2015 From: jim.rosenblum@REDACTED (Jim) Date: Fri, 20 Nov 2015 23:27:42 -0500 Subject: [erlang-questions] guard clause oddity, maybe bug (in my brain)? In-Reply-To: <2AA4A1FA-0F32-4B6D-94EB-4B5185A2ACD0@gmail.com> References: <08B13D4A-EFEE-412C-8BE2-180BE428F009@gmail.com> <2AA4A1FA-0F32-4B6D-94EB-4B5185A2ACD0@gmail.com> Message-ID: <735D054E-BAF6-4069-9845-B00144601FB7@gmail.com> Thank you both, I guess I don't fully understand the associative and precedence of these operators. Having said that it is surprising to me that (not is_tuple(Elt)) andalso ?IS_INDEX(I) can ever evaluate to true when Elt is a tuple. Andalso short circuits so it's curious to me that a fully parenthesized clause that evaluates to false to the left of the andalso wouldn't short circuit everything regardless of what happens to the right of the andalso. > On Nov 20, 2015, at 3:46 PM, Pierre Fenoll wrote: > > Dmitry is on point. > > 1. Remove the useless braces around not is_tuple... > 2. Add braces around ?IS_INDEX... > > If that fixes it, think about moving the braces inside the macro's definition. > >> On 20 Nov 2015, at 16:03, Dmitry Kolesnikov wrote: >> >> Hello, >> >> How does you macro IS_INDEX looks like? >> e.g. >> >> 5> A = {1}. >> {1} >> 6> (not is_tuple(A)) andalso is_number(A) orelse A > 1. >> true >> >> but >> >> 8> (not is_tuple(A)) andalso (is_number(A) orelse A > 1). >> false >> >> >> Best Regards, >> Dmitry >> >> >>> On Nov 20, 2015, at 4:03 PM, jim rosenblum wrote: >>> >>> I have a function: >>> >>> walk ([I|Is], [Elt|_]=Ts) when (not is_tuple(Elt)) andalso ?IS_INDEX(I) -> >>> Element = index_to_element(I,Ts), >>> case >>> ... snip ... >>> end. >>> >>> This function clause gets executed even though Elt is a tuple. >>> >>> I have a break point within this function clause. I am looking at the >>> debugger at the break-point within the clause, and I can plainly see that Elt >>> IS a tuple: within the debugger I can evaluate is_tuple(Elt) and it returns true, >>> I can evaluate (not is_tuple(Elt)) and correctly get FALSE. >>> >>> So.... how can I be in this clause when the guard evaluates to false? >>> >>> Thoughts? >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From kennethlakin@REDACTED Sat Nov 21 07:16:52 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Fri, 20 Nov 2015 22:16:52 -0800 Subject: [erlang-questions] guard clause oddity, maybe bug (in my brain)? In-Reply-To: <735D054E-BAF6-4069-9845-B00144601FB7@gmail.com> References: <08B13D4A-EFEE-412C-8BE2-180BE428F009@gmail.com> <2AA4A1FA-0F32-4B6D-94EB-4B5185A2ACD0@gmail.com> <735D054E-BAF6-4069-9845-B00144601FB7@gmail.com> Message-ID: <56500C54.5060007@gmail.com> On 11/20/2015 08:27 PM, Jim wrote: > > Thank you both, I guess I don't fully understand the associative and precedence of these operators. This seems a sensible way to look at it: 1> false andalso false orelse true. true 2> ((false andalso false) orelse true). true 3> (false andalso (false orelse true)). false andalso and orelse are described in the docs as Expr1 orelse Expr2 Expr1 andalso Expr2 You *have* to have two expressions for a given andalso/orelse, so you read expression 1 up there from left to right. First evaluate false andalso false (which is -obviously- false, and becomes the Expr1 of the next expression) Then evalutate false orelse true (which is true) Does that make sense? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ben@REDACTED Sat Nov 21 02:07:18 2015 From: ben@REDACTED (Benjamin Adams) Date: Fri, 20 Nov 2015 17:07:18 -0800 Subject: [erlang-questions] Calculate nanoseconds Message-ID: <1512794bfd4.fc1ad2d017127.401733681996823416@ethanstack.com> An HTML attachment was scrubbed... URL: From adam@REDACTED Fri Nov 20 21:18:04 2015 From: adam@REDACTED (Adam Cammack) Date: Fri, 20 Nov 2015 14:18:04 -0600 Subject: [erlang-questions] IP search optimization In-Reply-To: References: <20151119170510.GA72389@staff.retn.net> Message-ID: <20151120201804.GD8312@serenity> On Fri, Nov 20, 2015 at 06:01:50PM +0100, Sid Muller wrote: > > Looks more like a task for radix trees... > > Hmm, good point although with ets if my owner process gets killed I > inherit the ets and no data is lost. Not sure how that cold be done > with a radix tree. The ETS ordered_set table type is nearly as good for this purpose. > > The best you can achieve with ets is: > > on insert: make sure that inserted networks are normalized (do not have > > any bits set after prefix length). > > on lookup: > > 1. start with masklen of 32 > > 2. normalize address to current masklen > > 3. lookup for address/masklen record, return if found > > 4. if current masklen is zero - return 'not_found' > > 5. decrease masklen and repeat from step two. > > There are a few faster methods: > 1. Have a pre-scanned list of /CIDR [21, 20...], there are usually > about 10. But this still traverses the table 10 times in worst case. > > 2. Traverse the table with ets:lookup(ets:next(), this has the > advantage over #1 as it's much faster in most cases. > > 3. ets:select(MatchSpec), this is probably the fastest solution > because it should traverse the table only once. But right now I'm > having a bit of a problem with my select statement... see other email > [bitstring select] These are all slower. ets:match/{2,3}, ets:select/{2,3}, and repeatedly calling ets:next/2 should be avoided where possible on large tables because of their immense running time. CIDR blocks can be expressed as a tuple of two integers: the 32-bit padded mask and the number of significant bits. We can take advantage of Erlang's term ordering[1] and use the ordered_set table type with the block as the key {Mask, Length}. In an ordered_set table, ets:prev(Tab, {IP, 32}) will find the block that might contain that address[2]. Simple bit arithmetic will tell you if the address is in the block or not. This is about an order of magnitude faster on my box than doing ets:lookup/2 for increasingly less precise masks. Your issue with matching the bit strings in a match spec can be resolved by storing the bit strings as above and then using the bit arithmetic guard expressions (select/1 in the below gist), although I do not recommend this. When in doubt (and especially when certain), measure! https://gist.github.com/acammack/f631cf5ca978775f10f7 For 100 lookups on a 1,000,000 element table, Erlang/OTP 18.1: ets:prev/2 (ordered): 9.40000e-5 ets:select/3 (ordered): 1.45619e+1 ets:lookup/2 (ordered): 1.72300e-3 ets:lookup/2 (unordered): 7.94000e-4 [1] Tuples of the same length are ordered by their elements, starting with the first. [2] If individual addresses are stored in the same table, you will need to do an extra ets:lookup/2 for the address. If blocks can overlap, you will need to call ets:prev/2 until you get a block that does not match the address. Happy coding! -- Adam Cammack From felixgallo@REDACTED Sat Nov 21 08:23:27 2015 From: felixgallo@REDACTED (Felix Gallo) Date: Fri, 20 Nov 2015 23:23:27 -0800 Subject: [erlang-questions] Calculate nanoseconds In-Reply-To: <1512794bfd4.fc1ad2d017127.401733681996823416@ethanstack.com> References: <1512794bfd4.fc1ad2d017127.401733681996823416@ethanstack.com> Message-ID: http://www.erlang.org/doc/man/timer.html#tc-1 Is a good place to start. F. On Nov 20, 2015 11:18 PM, "Benjamin Adams" wrote: > Hello, > > How would I go about calculating Nanoseconds for a function call. Want to > use it for language profiling. > > Time.start() > Fun() ->...... > Receive > Io:format("nanoseconds: ~p", [Time.stop()]), > End. > > Something like that. > > ----- > Ben Adams > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob01@REDACTED Sat Nov 21 09:16:36 2015 From: jacob01@REDACTED (Jacob) Date: Sat, 21 Nov 2015 09:16:36 +0100 Subject: [erlang-questions] IP search optimization In-Reply-To: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> References: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> Message-ID: <56502864.9030008@gmx.net> Hi, On 21.11.2015 00:52, Geoff Cant wrote: > Hi Sid, are you writing an open source library for this functionality? I need exactly the same thing and was going to sit down to write it. > > With CIDR blocks, you can always calculate the lower and upper address in a range. > > [...] > > If you pre-convert your CIDR list into a range list, you can then select all the matching cidrs by > comparing Low =< IP, IP =< High, and returning the CIDR with the largest mask (if you do longest-prefix wins), > or the first in order if you?re doing an ordered CIDR match. > > Pre-converting CIDR -> Range will let you use ets:select - and might > be reasonably efficient in an ordered_set table if you?ve got large > numbers of CIDRs. I think if your ets key is {Low, High}, and it?s an > ordered_set table, the first result from a matching select will be > the one with the longest Mask anyway (as {<<192,168,0,0>>, > <<192,168,0,255>>} < {<<192,168,0,0>>, <<192,168,255,255>>} ? i.e. > 192.168.0.0/24 < 192.168.0.0/16 when converted to a range tuple). In that case you can just use ets:prev to get the network, as in {Low, High} = ets:prev(E, {IP, <<255,255,255,255,1>>}). and a check whether IP <= High holds. The <<255,255,255,255,1>> is an arbitrary value that just needs to be larger than any valid IP address, to make lookups like for IP = <<192,168,0,0>> reliable. If you have nested routes, you will have to call ets:prev({Low,High}) again until IP <= High or $end_of_table is returned. This can also be done without converting the IP adresses to binaries. The route specific data can also be put into the key to avoid an additional lookup as in E = ets:new(ip, [ordered_set]). ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}). ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}). ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}). {Low, High, If} = ets:prev(E, {{172,25,4,3},{256,256,256,256}, '_'}). -> {{172,16,0,0},{172,31,255,255},eth2} So a possible implementation is -define(LARGER_THAN_ANY_VALID_IP, <<-1:256>>). new() -> ets:new(?MODULE, [ordered_set]). addroute(E, Low, High, If) -> ets:insert(E, {{Low, High, If},1}). getroute(E, Ip) -> getroute(E, Ip, ets:prev(E, {Ip, ?LARGER_THAN_ANY_VALID_IP, '_'})). getroute(_E, Ip, {_Low, High, If}) when Ip =< High -> If; getroute(E, Ip, Key = {_Low, _High, _If}) -> getroute(E, Ip, ets:prev(E, Key)). which supports nested routes and would work with any kind of address encoding as long as the length is the same for all entries. Note that this will crash if there is no matching route, which can easily be avoided by adding a default route like in the example above. HTH, Jacob From jesper.louis.andersen@REDACTED Sat Nov 21 12:38:59 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 21 Nov 2015 12:38:59 +0100 Subject: [erlang-questions] Mersenne Twister In-Reply-To: References: Message-ID: You also have crypto:rand_bytes/1, crypto:strong_rand_bytes/1 and enacl:randombytes/1 if you need a CSPRNG. The rand module also makes sure it seeds randomly if it has not been seeded yet, which helps in a lot of situations where people accidentally forget to seed. The work here by Kenji, Dan and Sebastiano is pretty good. On Fri, Nov 20, 2015 at 11:53 PM, Felix Gallo wrote: > Aha, Jesper points out the rand namespace to me, which looks like it > provides a bunch more PRNGs as of R18. Thanks Jesper and, apparently, > Kenji et al. > > F. > > On Fri, Nov 20, 2015 at 2:46 PM, Felix Gallo wrote: > >> In 2011 there was discussion on this list around improving the stdlib >> random number generator, perhaps to join the crowd of languages using >> Mersenne Twister or a variant. >> >> As far as I can tell, that's about as far as anyone got; it appears that >> random:uniform() is still Wichmann-Hill, which is apparently held in low >> regard by randomness fans. >> >> Are there any extant plans to replace it with a more modern PRNG, such as >> MT? If they don't exist, what's the OTP team's current thoughts regarding >> the likelihood of accepting such a patch? >> >> F. >> >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nistrigunya@REDACTED Sat Nov 21 12:49:10 2015 From: nistrigunya@REDACTED (Avinash Dhumane) Date: Sat, 21 Nov 2015 17:19:10 +0530 Subject: [erlang-questions] spawn_link/2 behaviour Message-ID: Hello, I have following test program (on Windows): D:\Balin\Algo_v6\v0.5>type my_test.erl -module(my_test). -export([start/0]). start() -> register(?MODULE, spawn(fun() -> init(), local_loop() end)). init() -> spawn_link('feed@REDACTED', fun() -> register(?MODULE, self()), remote_loop() end), spawn('feed@REDACTED', fun() -> timer:sleep(timer:seconds(5)), % wait till the above process is registered ?MODULE ! hello, ?MODULE ! world, ?MODULE ! stop end). remote_loop() -> receive X -> {?MODULE, 'algo@REDACTED'} ! X, remote_loop() end. local_loop() -> receive stop -> io:format("stopped~n"); X -> io:format("~p~n", [X]), local_loop() end. D:\Balin\Algo_v6\v0.5> Then, I start 2 nodes, as follows: D:\Balin\Algo_v6\v0.5>start erl -name algo@REDACTED D:\Balin\Algo_v6\v0.5>start erl -name feed@REDACTED On the 'algo@REDACTED' node, I run the program as follows: Eshell V7.0 (abort with ^G) (algo@REDACTED)1> rpc:call('feed@REDACTED', erlang, whereis, [my_test]). undefined (algo@REDACTED)2> my_test:start(). true (algo@REDACTED)3> hello (algo@REDACTED)3> world (algo@REDACTED)3> stopped (algo@REDACTED)3> rpc:call('feed@REDACTED', erlang, whereis, [my_test]). <6755.43.0> (algo@REDACTED)4> The messaging works as I expected. However, the process registered as 'my_test' on the node 'feed@REDACTED' does not terminate. I expected it to terminate since I started it using spawn_link/2 from the calling process on 'algo@REDACTED' which terminates after receiving the 'stop' message. It keeps running, as is observed on the node 'feed@REDACTED': Eshell V7.0 (abort with ^G) (feed@REDACTED)1> whereis(my_test). <0.43.0> (feed@REDACTED)2> What is wrong with the setup done in the test program above that the linked process on another node does not terminate? Please advise. Thank you. Avinash -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sat Nov 21 16:23:38 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 21 Nov 2015 16:23:38 +0100 Subject: [erlang-questions] Calculate nanoseconds In-Reply-To: References: <1512794bfd4.fc1ad2d017127.401733681996823416@ethanstack.com> Message-ID: You can adapt the source code of timer:tc/1 to use nano_seconds timing rather than microseconds, but I would be weary you can get accurate timing at the nanosecond scale by running a single experiment. You would have to run more rounds and divide in order to figure out what a real timing would be. Also, you should collect many samples, and do bootstrapping on the dataset in order to figure out if the average is a stable number, or if it is likely to have been perturbed by something. On Sat, Nov 21, 2015 at 8:23 AM, Felix Gallo wrote: > http://www.erlang.org/doc/man/timer.html#tc-1 > > Is a good place to start. > > F. > On Nov 20, 2015 11:18 PM, "Benjamin Adams" wrote: > >> Hello, >> >> How would I go about calculating Nanoseconds for a function call. Want >> to use it for language profiling. >> >> Time.start() >> Fun() ->...... >> Receive >> Io:format("nanoseconds: ~p", [Time.stop()]), >> End. >> >> Something like that. >> >> ----- >> Ben Adams >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.rosenblum@REDACTED Sat Nov 21 16:55:27 2015 From: jim.rosenblum@REDACTED (Jim) Date: Sat, 21 Nov 2015 10:55:27 -0500 Subject: [erlang-questions] erlang-questions Digest, Vol 244, Issue 8 In-Reply-To: References: Message-ID: Perfect - that fixed my head. Thanks. Date: Fri, 20 Nov 2015 22:16:52 -0800 From: Kenneth Lakin To: erlang-questions@REDACTED Subject: Re: [erlang-questions] guard clause oddity, maybe bug (in my brain)? Message-ID: <56500C54.5060007@REDACTED> Content-Type: text/plain; charset="windows-1252" This seems a sensible way to look at it: 1> false andalso false orelse true. true 2> ((false andalso false) orelse true). true 3> (false andalso (false orelse true)). false andalso and orelse are described in the docs as Expr1 orelse Expr2 Expr1 andalso Expr2 You *have* to have two expressions for a given andalso/orelse, so you read expression 1 up there from left to right. First evaluate false andalso false (which is -obviously- false, and becomes the Expr1 of the next expression) Then evalutate false orelse true (which is true) Does that make sense? Sent from my iPad > On Nov 21, 2015, at 6:00 AM, erlang-questions-request@REDACTED wrote: > > Date: Fri, 20 Nov 2015 22:16:52 -0800 > From: Kenneth Lakin > To: erlang-questions@REDACTED > Subject: Re: [erlang-questions] guard clause oddity, maybe bug (in my > brain)? > Message-ID: <56500C54.5060007@REDACTED> > Content-Type: text/plain; charset="windows-1252" > >> On 11/20/2015 08:27 PM, Jim wrote: >> >> Thank you both, I guess I don't fully understand the associative and precedence of these operators. > > This seems a sensible way to look at it: > > 1> false andalso false orelse true. > true > 2> ((false andalso false) orelse true). > true > 3> (false andalso (false orelse true)). > false > > andalso and orelse are described in the docs as > > Expr1 orelse Expr2 > Expr1 andalso Expr2 > > You *have* to have two expressions for a given andalso/orelse, so you > read expression 1 up there from left to right. > First evaluate > false andalso false (which is -obviously- false, and becomes the Expr1 > of the next expression) > Then evalutate > false orelse true (which is true) > > Does that make sense? -------------- next part -------------- An HTML attachment was scrubbed... URL: From montuori@REDACTED Sat Nov 21 20:04:09 2015 From: montuori@REDACTED (Kevin Montuori) Date: Sat, 21 Nov 2015 13:04:09 -0600 Subject: [erlang-questions] Erlang and Docker In-Reply-To: (David Goehrig's message of "Mon, 22 Jun 2015 10:05:52 -0400") References: <3D01227D-7F2B-43BA-81EF-9D8B381C5F92@vailsys.com> Message-ID: >>>>> "dg" == David Goehrig writes: dg> At work, we've been deploying a wide range of Erlang apps on dg> Docker. I'm working on a blog post ( and presentation ) on how dg> we manage our containers across multiple cloud providers. Hi David -- I realize this is digging up a post from nearly six months ago but I'm wondering if this blog post materialized? Or if anyone out there had new thoughts on distributed Erlang and Docker? Thanks! k. -- Kevin Montuori montuori@REDACTED From sid5@REDACTED Sat Nov 21 20:55:58 2015 From: sid5@REDACTED (Sid Muller) Date: Sat, 21 Nov 2015 20:55:58 +0100 Subject: [erlang-questions] IP search optimization In-Reply-To: <20151120201804.GD8312@serenity> References: <20151119170510.GA72389@staff.retn.net> , <20151120201804.GD8312@serenity> Message-ID: Adam, that ets:lookup() via ets:prev() is actually brilliant!!! Thank you! > Sent: Friday, November 20, 2015 at 12:18 PM > From: "Adam Cammack" > To: "Sid Muller" > Cc: erlang-questions > Subject: Re: [erlang-questions] IP search optimization > > On Fri, Nov 20, 2015 at 06:01:50PM +0100, Sid Muller wrote: > > > Looks more like a task for radix trees... > > > > Hmm, good point although with ets if my owner process gets killed I > > inherit the ets and no data is lost. Not sure how that cold be done > > with a radix tree. > > The ETS ordered_set table type is nearly as good for this purpose. > > > > The best you can achieve with ets is: > > > on insert: make sure that inserted networks are normalized (do not have > > > any bits set after prefix length). > > > on lookup: > > > 1. start with masklen of 32 > > > 2. normalize address to current masklen > > > 3. lookup for address/masklen record, return if found > > > 4. if current masklen is zero - return 'not_found' > > > 5. decrease masklen and repeat from step two. > > > > There are a few faster methods: > > 1. Have a pre-scanned list of /CIDR [21, 20...], there are usually > > about 10. But this still traverses the table 10 times in worst case. > > > > 2. Traverse the table with ets:lookup(ets:next(), this has the > > advantage over #1 as it's much faster in most cases. > > > > 3. ets:select(MatchSpec), this is probably the fastest solution > > because it should traverse the table only once. But right now I'm > > having a bit of a problem with my select statement... see other email > > [bitstring select] > > These are all slower. ets:match/{2,3}, ets:select/{2,3}, and repeatedly > calling ets:next/2 should be avoided where possible on large tables > because of their immense running time. > > CIDR blocks can be expressed as a tuple of two integers: the 32-bit > padded mask and the number of significant bits. We can take advantage of > Erlang's term ordering[1] and use the ordered_set table type with the > block as the key {Mask, Length}. In an ordered_set table, ets:prev(Tab, > {IP, 32}) will find the block that might contain that address[2]. Simple > bit arithmetic will tell you if the address is in the block or not. This > is about an order of magnitude faster on my box than doing ets:lookup/2 > for increasingly less precise masks. > > Your issue with matching the bit strings in a match spec can be resolved > by storing the bit strings as above and then using the bit arithmetic > guard expressions (select/1 in the below gist), although I do not > recommend this. > > When in doubt (and especially when certain), measure! > > https://gist.github.com/acammack/f631cf5ca978775f10f7 > > For 100 lookups on a 1,000,000 element table, Erlang/OTP 18.1: > > ets:prev/2 (ordered): 9.40000e-5 > ets:select/3 (ordered): 1.45619e+1 > ets:lookup/2 (ordered): 1.72300e-3 > ets:lookup/2 (unordered): 7.94000e-4 > > [1] Tuples of the same length are ordered by their elements, starting > with the first. > [2] If individual addresses are stored in the same table, you will need > to do an extra ets:lookup/2 for the address. If blocks can overlap, > you will need to call ets:prev/2 until you get a block that does not > match the address. > > Happy coding! > > -- > Adam Cammack > From sid5@REDACTED Sat Nov 21 20:59:57 2015 From: sid5@REDACTED (Sid Muller) Date: Sat, 21 Nov 2015 20:59:57 +0100 Subject: [erlang-questions] IP search optimization In-Reply-To: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> References: , <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> Message-ID: > Sent: Friday, November 20, 2015 at 3:52 PM > From: "Geoff Cant" > To: "Sid Muller" > Cc: Erlang > Subject: Re: [erlang-questions] IP search optimization > > Hi Sid, are you writing an open source library for this functionality? I need exactly the same thing and was going to sit down to write it. > > With CIDR blocks, you can always calculate the lower and upper address in a range. > > > -type ip4() :: <<_:32/bits>>. > > -type ip4_mask() :: 0..32. > > -type ip4_cidr() :: {ip4(), ip4_mask()}. > > -spec to_range(ip4_cidr()) -> {Low::ip4(), High::ip4()}. > > to_range({<>, Mask}) > > when 0 =< Mask, > > Mask =< 32 -> > > HostBits = 32 - Mask, > > {<>, > > <>}. > > If you pre-convert your CIDR list into a range list, you can then select all the matching cidrs by comparing Low =< IP, IP =< High, and returning the CIDR with the largest mask (if you do longest-prefix wins), or the first in order if you?re doing an ordered CIDR match. > > Pre-converting CIDR -> Range will let you use ets:select - and might be reasonably efficient in an ordered_set table if you?ve got large numbers of CIDRs. I think if your ets key is {Low, High}, and it?s an ordered_set table, the first result from a matching select will be the one with the longest Mask anyway (as {<<192,168,0,0>>, <<192,168,0,255>>} < {<<192,168,0,0>>, <<192,168,255,255>>} ? i.e. 192.168.0.0/24 < 192.168.0.0/16 when converted to a range tuple). > > > IP = something, > > Matching = ets:select(cidr_table, ets:fun2ms(fun (#record{range={Low, High}}) when Low =< IP, IP =< High) -> object() end)). > > > Anyway - I?m either going to write something like this in the next couple of weeks, or use yours if it?s public and suitable :) Hi Geoff, check out Adam's response, the 9 liner prev() does the job really well. From felixgallo@REDACTED Sat Nov 21 21:52:15 2015 From: felixgallo@REDACTED (Felix Gallo) Date: Sat, 21 Nov 2015 12:52:15 -0800 Subject: [erlang-questions] Erlang and Docker In-Reply-To: References: <3D01227D-7F2B-43BA-81EF-9D8B381C5F92@vailsys.com> Message-ID: Erlang is already like a miniaturized operating system implementing well-scheduled lightweight microservices, and releases ( http://www.erlang.org/doc/design_principles/release_structure.html) are already a redistributable, self-contained, minimizable unit of deployment that can be monitored, inspected and logged using standard unix workflow. In my opinion, the cost-benefit of using docker is highly questionable even in its optimal use case, where the dev team is composed entirely of inexperienced front end javascript developers trying to deploy monolithic microservices and pretending they're being lightweight. Maybe there's a case to be made for tooling which takes languages with terrible manageability and hides them behind something, but it's not at all clear that docker is the answer to that or any other question. It makes a negative amount of sense to layer the highly ceremonial, unperformant, problem-multiplying docker abstraction on top of an actually light, already existing microservices architecture, and I wouldn't wish ownership of the resulting mutant chimera on my worst enemy. But again -- my personal opinion. F. On Sat, Nov 21, 2015 at 11:04 AM, Kevin Montuori wrote: > >>>>> "dg" == David Goehrig writes: > > dg> At work, we've been deploying a wide range of Erlang apps on > dg> Docker. I'm working on a blog post ( and presentation ) on how > dg> we manage our containers across multiple cloud providers. > > Hi David -- > > I realize this is digging up a post from nearly six months ago but I'm > wondering if this blog post materialized? Or if anyone out there had > new thoughts on distributed Erlang and Docker? > > Thanks! > k. > > > -- > Kevin Montuori > montuori@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From radek@REDACTED Sun Nov 22 00:36:25 2015 From: radek@REDACTED (Rad Gruchalski) Date: Sun, 22 Nov 2015 00:36:25 +0100 Subject: [erlang-questions] Erlang and Docker In-Reply-To: References: <3D01227D-7F2B-43BA-81EF-9D8B381C5F92@vailsys.com> Message-ID: <288678AD71A746DCBAFCBDA67CD8D096@gruchalski.com> Nevertheless, it would be nice to read the article. Having been thinking about it, Erlang is the interesting unicorn case. Still, I?d like to read what the reasoning was. Kind regards, Radek Gruchalski radek@REDACTED (mailto:radek@REDACTED) (mailto:radek@REDACTED) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Saturday, 21 November 2015 at 21:52, Felix Gallo wrote: > Erlang is already like a miniaturized operating system implementing well-scheduled lightweight microservices, and releases (http://www.erlang.org/doc/design_principles/release_structure.html) are already a redistributable, self-contained, minimizable unit of deployment that can be monitored, inspected and logged using standard unix workflow. > > In my opinion, the cost-benefit of using docker is highly questionable even in its optimal use case, where the dev team is composed entirely of inexperienced front end javascript developers trying to deploy monolithic microservices and pretending they're being lightweight. Maybe there's a case to be made for tooling which takes languages with terrible manageability and hides them behind something, but it's not at all clear that docker is the answer to that or any other question. It makes a negative amount of sense to layer the highly ceremonial, unperformant, problem-multiplying docker abstraction on top of an actually light, already existing microservices architecture, and I wouldn't wish ownership of the resulting mutant chimera on my worst enemy. But again -- my personal opinion. > > F. > > On Sat, Nov 21, 2015 at 11:04 AM, Kevin Montuori wrote: > > >>>>> "dg" == David Goehrig writes: > > > > dg> At work, we've been deploying a wide range of Erlang apps on > > dg> Docker. I'm working on a blog post ( and presentation ) on how > > dg> we manage our containers across multiple cloud providers. > > > > Hi David -- > > > > I realize this is digging up a post from nearly six months ago but I'm > > wondering if this blog post materialized? Or if anyone out there had > > new thoughts on distributed Erlang and Docker? > > > > Thanks! > > k. > > > > > > -- > > Kevin Montuori > > montuori@REDACTED (mailto:montuori@REDACTED) > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Sun Nov 22 11:19:45 2015 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Sun, 22 Nov 2015 11:19:45 +0100 Subject: [erlang-questions] Erlang and Docker In-Reply-To: References: <3D01227D-7F2B-43BA-81EF-9D8B381C5F92@vailsys.com> Message-ID: <1F3166A3-6DC6-4DC0-A772-455602CB7E47@gmail.com> > 21 nov. 2015 kl. 21:52 skrev Felix Gallo : > > Erlang is already like a miniaturized operating system implementing well-scheduled lightweight microservices, and releases (http://www.erlang.org/doc/design_principles/release_structure.html) are already a redistributable, self-contained, minimizable unit of deployment that can be monitored, inspected and logged using standard unix workflow. > > In my opinion, the cost-benefit of using docker is highly questionable even in its optimal use case, where the dev team is composed entirely of inexperienced front end javascript developers trying to deploy monolithic microservices and pretending they're being lightweight. Maybe there's a case to be made for tooling which takes languages with terrible manageability and hides them behind something, but it's not at all clear that docker is the answer to that or any other question. It makes a negative amount of sense to layer the highly ceremonial, unperformant, problem-multiplying docker abstraction on top of an actually light, already existing microservices architecture, and I wouldn't wish ownership of the resulting mutant chimera on my worst enemy. But again -- my personal opinion. > > F. Perhaps this is true for homogeneous systems such as systems only built with Erlang components. Many systems are however hybrid systems with different component types and a container architecture (such as docker) could help in making things more homogeneous. Stefan > >> On Sat, Nov 21, 2015 at 11:04 AM, Kevin Montuori wrote: >> >>>>> "dg" == David Goehrig writes: >> >> dg> At work, we've been deploying a wide range of Erlang apps on >> dg> Docker. I'm working on a blog post ( and presentation ) on how >> dg> we manage our containers across multiple cloud providers. >> >> Hi David -- >> >> I realize this is digging up a post from nearly six months ago but I'm >> wondering if this blog post materialized? Or if anyone out there had >> new thoughts on distributed Erlang and Docker? >> >> Thanks! >> k. >> >> >> -- >> Kevin Montuori >> montuori@REDACTED >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Sun Nov 22 12:38:45 2015 From: zxq9@REDACTED (zxq9) Date: Sun, 22 Nov 2015 20:38:45 +0900 Subject: [erlang-questions] Erlang and Docker In-Reply-To: <1F3166A3-6DC6-4DC0-A772-455602CB7E47@gmail.com> References: <1F3166A3-6DC6-4DC0-A772-455602CB7E47@gmail.com> Message-ID: <1957652.MvnmcM9e3M@changa> On 2015?11?22? ??? 11:19:45 Stefan Hellkvist wrote: > > 21 nov. 2015 kl. 21:52 skrev Felix Gallo : > > Erlang is already like a miniaturized operating system implementing well-scheduled lightweight microservices, and releases (http://www.erlang.org/doc/design_principles/release_structure.html) are already a redistributable, self-contained, minimizable unit of deployment that can be monitored, inspected and logged using standard unix workflow. > > > > In my opinion, the cost-benefit of using docker is highly questionable even in its optimal use case, where the dev team is composed entirely of inexperienced front end javascript developers trying to deploy monolithic microservices and pretending they're being lightweight. Maybe there's a case to be made for tooling which takes languages with terrible manageability and hides them behind something, but it's not at all clear that docker is the answer to that or any other question. It makes a negative amount of sense to layer the highly ceremonial, unperformant, problem-multiplying docker abstraction on top of an actually light, already existing microservices architecture, and I wouldn't wish ownership of the resulting mutant chimera on my worst enemy. But again -- my personal opinion. > > > > F. > > Perhaps this is true for homogeneous systems such as systems only built with Erlang components. Many systems are however hybrid systems with different component types and a container architecture (such as docker) could help in making things more homogeneous. [BEGIN RANT: Docker is not a best practice] Then those systems have not either been appropriately bundled (lots of releases drop other language bundles in priv/) or abstracted across the network as separate services (this is SO FREAKING NOT HARD TO DO AND MAKES LIFE EASIER FOREVER WHY ISN'T THIS THE DEFAULT MODE OF THOUGHT WTF AHHHHHHH!). The reason I've grown skeptical of both docker and "fire and forget" virtualization is that I generally see containerization as a way for devops to barely cobble an environment together in a rush before some deadline, slap a label like "WorkingBaseSystem X" on it -- and consider their job with that particular platform as complete, forever. Which means... - It is never maintained (until something like heartbleed... dhoh!). - How it was made to work in the first place is lightly/never documented. - The interactions among components are not documented. - Interfaces are not defined. - Sloppy hacks are very hard to discover. - Updating the system is a *terrifying* prospect. - Reviewing the old code to document structure/flow becomes archaeology. They become security landmines until the end of time, and most management teams seem to think "its not broke; don't fix it", but are confused as to why their devs becomes paralyzed the moment some new feature or bugfix is required in the old system -- so instead begin replicating functionality buried in the old system. In the case where the containerized system represents the core business, sure, that is maintained and updated -- and in those cases containerization isn't really much of an asset because its not competing with other configurations for primacy over settings. In the case where the containerized system is not the core business containerization encourages the horrible trends I wrote about above. In the bad case this means that there will probably never *be* another major version release of whatever system lives in the "hybrid system with different component types and a container architecture" because nobody will ever be able to figure either what that system does inside, or never figure out how to resolve the christmas-lights-tangle of settings and version choices that has been containerized. If the system is well maintained, these things are obvious and docker isn't buying you much; if the system is not well maintained docker is just covering up the accumulating stench by dousing it in fabreeze and wrapping it in a blanket really tight. With regard specifically to Erlang, a release basically *is* a container, though placing that again within a VM/docker/container/whatever doesn't really hurt anything -- its just not as big a win, and it certainly does make things more complex for no reason (now you have to make two releases, one nested within the other). "But nobody has time to [do X]" when X :: document the mysterious system : define interfaces : figure out what versions of what work with what else : update component subsystems : whatever other time-consuming task Then you - Make time - Appreciate that you are going to swim in mud until you make enough money to break into open water - Start looking for another job before your current company implodes Theoretically maybe docker has some really super great benefits for Erlang... but in practice I've generally just seen it (and not just docker, again, all forms of containerization) become an excuse for (and later evolve into a justification of) profoundly sloppy operational practices. For some reason I see this a bit less with VM-based solutions than docker (but seen it there, too). "Gee, Craig, why all the vinegar?" you may ask. Because the difference between a heterogeneous/homogeneous environment should not be a major design issue, which implies that methods to deal with different components as totally separate, totally abstracted entities should be a fundamental part of any large system design (this method is usually called "sockets", and they don't all have to be HTTP... >gasp!<). You shouldn't be running gleefully along and just suddenly realize you've built not just a monolithic code base, but a Jenga game of the supporting system that it runs on as well. This is a solution to a problem that shouldn't exist, and the way to do better things is not to just sweep it under the rug and consider this "a best practice", but rather regard the need for this form of deployment as a sign that you've screwed up and painted yourself into a corner. If you find yourself having to use docker, especially for an Erlang-based system, you should think "thank goodness we can bundle this awful mess up for now until we get our fundamental problems sorted out". [END RANT: DINABP] -Craig From t@REDACTED Sun Nov 22 14:45:42 2015 From: t@REDACTED (Tristan Sloughter) Date: Sun, 22 Nov 2015 07:45:42 -0600 Subject: [erlang-questions] Erlang and Docker In-Reply-To: <1957652.MvnmcM9e3M@changa> References: <1F3166A3-6DC6-4DC0-A772-455602CB7E47@gmail.com> <1957652.MvnmcM9e3M@changa> Message-ID: <1448199942.2711634.446647953.171EC39A@webmail.messagingengine.com> I have to say that I partially agree with those here pointing out the use of a target system providing you with many similar benefits to the use of a docker image. And I'm no docker fan, but the argument that a release is basically a container is simply false and scrubs away all the useful purposes of a container, jail, zone, etc. A release is not a replacement for the cases you need a cgroup, jail or zone. But if the only reason you think you need such a thing is to start an Erlang node you may be engaging in overkill that will be technical debt. -- Tristan Sloughter t@REDACTED From zxq9@REDACTED Sun Nov 22 15:22:27 2015 From: zxq9@REDACTED (zxq9) Date: Sun, 22 Nov 2015 23:22:27 +0900 Subject: [erlang-questions] Erlang and Docker In-Reply-To: <1448199942.2711634.446647953.171EC39A@webmail.messagingengine.com> References: <1957652.MvnmcM9e3M@changa> <1448199942.2711634.446647953.171EC39A@webmail.messagingengine.com> Message-ID: <1778791.TdEkoO8Abt@changa> On 2015?11?22? ??? 07:45:42 Tristan Sloughter wrote: > I have to say that I partially agree with those here pointing out the > use of a target system providing you with many similar benefits to the > use of a docker image. And I'm no docker fan, but the argument that a > release is basically a container is simply false and scrubs away all the > useful purposes of a container, jail, zone, etc. > > A release is not a replacement for the cases you need a cgroup, jail or > zone. But if the only reason you think you need such a thing is to start > an Erlang node you may be engaging in overkill that will be technical > debt. Absolutely. To expand the discussion a bit further though... I've seen more folks blindly employ segregation facilities like cgoups and especially jails because they thought they were primary security features than realizing they are environmental segregations. "Scrubs away all the useful purposes of a container, jail, zone, etc." I don't find these particularly useful compared to simple virtualization -- and virtualization itself is also not a primary security feature. (The term "jail" was poorly chosen, in retrospect.) With this in mind, I tend to view things rather more black and white: VMs or local execution -- pretty much anything else is indeed technical debt. But its the cool-feeling kind of technical debt that comes complete with a bodyguard of totally kickass websites, buzzwords and ready-made consultant army. (otoh, if you have to pay a fee per OS image or committed core, then VMs can easily move you from "technical" to "finanical" debt) I'm not saying these things don't work -- they do. But they have never actually made deployment or maintenance any easier over the long term -- instead merely move the complexity involved in those tasks elsewhere for a while. I wind up having to learn and *maintain* several new skills on top of >gasp< already having to deploy in VMs anyway. Splitting system components up as network services, though -- that actually *has* been a big win, and I don't really care how those are deployed, what language they are written in, or how they are hosted. Containerization, of any form, encourages coupling at the grand architecture level because it hides the long-term pain of coupling tradeoffs very well. That is really what I'm most concerned about. A well written service should be deployable to any target platform in a simple way. We just skip this step when we aren't writing client-side software -- but its not so hard to get right if we start out with this in mind instead of just barely getting a development environment working and then letting devops make that the blueprint for production. Here I am certainly not talking just about Erlang-only services -- it is a broad problem. I'm not particularly fond of Erlang releases for *everything* -- I think it is very often overkill and understanding what a release even is seems to repel a lot of folks who like the language and runtime but don't grok that aspect of the environment (after all, most Erlang code written today is not targetting embedded phone switches with constrained resources and limited underlying system facilities), but it *does* simplify deployment dramatically (once you come to terms with the technical debt involved in getting Erlang releases to work right...), and ease-of-deployment seems to be the focus of most chatter about containerization. -Craig From snar@REDACTED Sun Nov 22 17:53:45 2015 From: snar@REDACTED (Alexandre Snarskii) Date: Sun, 22 Nov 2015 19:53:45 +0300 Subject: [erlang-questions] IP search optimization In-Reply-To: <20151120201804.GD8312@serenity> References: <20151119170510.GA72389@staff.retn.net> <20151120201804.GD8312@serenity> Message-ID: <20151122165345.GA13944@staff.retn.net> On Fri, Nov 20, 2015 at 02:18:04PM -0600, Adam Cammack wrote: > On Fri, Nov 20, 2015 at 06:01:50PM +0100, Sid Muller wrote: > > > Looks more like a task for radix trees... > > > > Hmm, good point although with ets if my owner process gets killed I > > inherit the ets and no data is lost. Not sure how that cold be done > > with a radix tree. > > The ETS ordered_set table type is nearly as good for this purpose. > > > > The best you can achieve with ets is: > > > on insert: make sure that inserted networks are normalized (do not have > > > any bits set after prefix length). > > > on lookup: > > > 1. start with masklen of 32 > > > 2. normalize address to current masklen > > > 3. lookup for address/masklen record, return if found > > > 4. if current masklen is zero - return 'not_found' > > > 5. decrease masklen and repeat from step two. > > > > There are a few faster methods: > > 1. Have a pre-scanned list of /CIDR [21, 20...], there are usually > > about 10. But this still traverses the table 10 times in worst case. > > > > 2. Traverse the table with ets:lookup(ets:next(), this has the > > advantage over #1 as it's much faster in most cases. > > > > 3. ets:select(MatchSpec), this is probably the fastest solution > > because it should traverse the table only once. But right now I'm > > having a bit of a problem with my select statement... see other email > > [bitstring select] > > These are all slower. ets:match/{2,3}, ets:select/{2,3}, and repeatedly > calling ets:next/2 should be avoided where possible on large tables > because of their immense running time. > > CIDR blocks can be expressed as a tuple of two integers: the 32-bit > padded mask and the number of significant bits. We can take advantage of > Erlang's term ordering[1] and use the ordered_set table type with the > block as the key {Mask, Length}. In an ordered_set table, ets:prev(Tab, > {IP, 32}) will find the block that might contain that address[2]. Simple > bit arithmetic will tell you if the address is in the block or not. This > is about an order of magnitude faster on my box than doing ets:lookup/2 > for increasingly less precise masks. This algorithm is good for your test set of non-overlapping networks, but not in general. Let's assume we add entry {0, 0} (default route) to your set and try to lookup some non-existant route. In case we look for 244.42.192.0/20 - ets:prev returns 244.42.64.0/20 (last entry in your set, which does not match), ets:prev for 244.42.64.0/20 returns 244.42.48.0/20 (does not match again), then .32.0/20 and only after ONE MILLION lookups you get matching 0.0.0.0/0 which is the right answer. With decrementing masklen you will get the same result in 32 lookups. > Your issue with matching the bit strings in a match spec can be resolved > by storing the bit strings as above and then using the bit arithmetic > guard expressions (select/1 in the below gist), although I do not > recommend this. > > When in doubt (and especially when certain), measure! > > https://gist.github.com/acammack/f631cf5ca978775f10f7 > > For 100 lookups on a 1,000,000 element table, Erlang/OTP 18.1: > > ets:prev/2 (ordered): 9.40000e-5 > ets:select/3 (ordered): 1.45619e+1 > ets:lookup/2 (ordered): 1.72300e-3 > ets:lookup/2 (unordered): 7.94000e-4 > > [1] Tuples of the same length are ordered by their elements, starting > with the first. > [2] If individual addresses are stored in the same table, you will need > to do an extra ets:lookup/2 for the address. If blocks can overlap, > you will need to call ets:prev/2 until you get a block that does not > match the address. > > Happy coding! > > -- > Adam Cammack > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From snar@REDACTED Sun Nov 22 18:11:59 2015 From: snar@REDACTED (Alexandre Snarskii) Date: Sun, 22 Nov 2015 20:11:59 +0300 Subject: [erlang-questions] IP search optimization In-Reply-To: <56502864.9030008@gmx.net> References: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> <56502864.9030008@gmx.net> Message-ID: <20151122171158.GB13944@staff.retn.net> On Sat, Nov 21, 2015 at 09:16:36AM +0100, Jacob wrote: > Hi, > > On 21.11.2015 00:52, Geoff Cant wrote: > > Hi Sid, are you writing an open source library for this functionality? I need exactly the same thing and was going to sit down to write it. > > > > With CIDR blocks, you can always calculate the lower and upper address in a range. > > > > [...] > > > > If you pre-convert your CIDR list into a range list, you can then select all the matching cidrs by > > comparing Low =< IP, IP =< High, and returning the CIDR with the > largest mask (if you do longest-prefix wins), > > or the first in order if you?re doing an ordered CIDR match. > > > > Pre-converting CIDR -> Range will let you use ets:select - and might > > be reasonably efficient in an ordered_set table if you?ve got large > > numbers of CIDRs. I think if your ets key is {Low, High}, and it?s an > > ordered_set table, the first result from a matching select will be > > the one with the longest Mask anyway (as {<<192,168,0,0>>, > > <<192,168,0,255>>} < {<<192,168,0,0>>, <<192,168,255,255>>} ? i.e. > > 192.168.0.0/24 < 192.168.0.0/16 when converted to a range tuple). > > In that case you can just use ets:prev to get the network, as in > > {Low, High} = ets:prev(E, {IP, <<255,255,255,255,1>>}). > > and a check whether IP <= High holds. The <<255,255,255,255,1>> is an > arbitrary value that just needs to be larger than any valid IP address, > to make lookups like for IP = <<192,168,0,0>> reliable. If you have > nested routes, you will have to call ets:prev({Low,High}) again until IP > <= High or $end_of_table is returned. > > This can also be done without converting the IP adresses to binaries. > The route specific data can also be put into the key to avoid an > additional lookup as in > > E = ets:new(ip, [ordered_set]). > ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}). > ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}). > ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}). > > {Low, High, If} = ets:prev(E, {{172,25,4,3},{256,256,256,256}, '_'}). > -> {{172,16,0,0},{172,31,255,255},eth2} E = ets:new(ip, [ordered_set]), ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}), ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}), ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}), ets:insert(E, {{{192,168,1,0},{192,168,1,255},eth3},1}), {Low, Hi, If} = ets:prev(E, {{192,168,10,11}, {256,256,256,256}, '_'}). {{192,168,1,0},{192,168,1,255},eth3} As you see, lookup for 192.168.10.11 returns incorrect network 192.168.1.0/24, not 192.168.0.0/16. Another example with the same setup: ets:prev(E, {{239,0,0,1}, {256,256,256,256}, '_'}). {{192,168,1,0},{192,168,1,255},eth3} lookup for 239.0.0.1 returns 192.168.1.0/24, not 0.0.0.0/0 as it should. From montuori@REDACTED Sun Nov 22 18:27:47 2015 From: montuori@REDACTED (Kevin Montuori) Date: Sun, 22 Nov 2015 11:27:47 -0600 Subject: [erlang-questions] Erlang and Docker In-Reply-To: <1778791.TdEkoO8Abt@changa> (zxq9@zxq9.com's message of "Sun, 22 Nov 2015 23:22:27 +0900") References: <1957652.MvnmcM9e3M@changa> <1448199942.2711634.446647953.171EC39A@webmail.messagingengine.com> <1778791.TdEkoO8Abt@changa> Message-ID: >>>>> "z" == zxq9 writes: z> I'm not saying these things don't work -- they do. But they have z> never actually made deployment or maintenance any easier over the z> long term -- instead merely move the complexity involved in those z> tasks elsewhere for a while. Bringing up Docker certainly roused up some passions. I'm not sold on containers as the solution to the world's ills but I'm also not willing to eschew their use just because they're not always ideal. Right tool for the job and all that. Anyhow, I apologize for not being specific with the question I asked. I have a use case where containers are indicated and need a very small distributed cache. Mnesia's my first choice but only if I don't have to figure out the Erlang-in-Docker clustering myself; I was hoping someone might have insight into (or a recipe for!) making that work. If not there are other easily implementable solutions that I'll investigate. Thanks all ... the discussion has made interesting reading! k. -- Kevin Montuori montuori@REDACTED From jacob01@REDACTED Sun Nov 22 19:02:35 2015 From: jacob01@REDACTED (Jacob) Date: Sun, 22 Nov 2015 19:02:35 +0100 Subject: [erlang-questions] IP search optimization In-Reply-To: <20151122171158.GB13944@staff.retn.net> References: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> <56502864.9030008@gmx.net> <20151122171158.GB13944@staff.retn.net> Message-ID: <5652033B.1030108@gmx.net> Hi Alexandre, > E = ets:new(ip, [ordered_set]), > ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}), > ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}), > ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}), > ets:insert(E, {{{192,168,1,0},{192,168,1,255},eth3},1}), > {Low, Hi, If} = ets:prev(E, {{192,168,10,11}, {256,256,256,256}, '_'}). > {{192,168,1,0},{192,168,1,255},eth3} > > As you see, lookup for 192.168.10.11 returns incorrect network 192.168.1.0/24, > not 192.168.0.0/16. Another example with the same setup: > > ets:prev(E, {{239,0,0,1}, {256,256,256,256}, '_'}). > {{192,168,1,0},{192,168,1,255},eth3} > > lookup for 239.0.0.1 returns 192.168.1.0/24, not 0.0.0.0/0 as it should. > That is why I wrote On 22.11.2015 18:11, Alexandre Snarskii wrote: > On Sat, Nov 21, 2015 at 09:16:36AM +0100, Jacob wrote: >> >> In that case you can just use ets:prev to get the network, as in >> >> {Low, High} = ets:prev(E, {IP, <<255,255,255,255,1>>}). >> That is crucial here: >> and a check whether IP <= High holds. Since {239,0,0,1} > {192,168,1,255} we are not done yet. >> The <<255,255,255,255,1>> is an >> arbitrary value that just needs to be larger than any valid IP address, >> to make lookups like for IP = <<192,168,0,0>> reliable. And further more: >> If you have >> nested routes, you will have to call ets:prev({Low,High}) again until IP >> <= High or $end_of_table is returned. Which tells how to get the correct result: 3> ets:prev(E, {{192,168,1,0},{192,168,1,255},eth3}). {{192,168,0,0},{192,168,255,255},eth1} 4> ets:prev(E, {{192,168,0,0},{192,168,255,255},eth1}). {{172,16,0,0},{172,31,255,255},eth2} 5> ets:prev(E, {{172,16,0,0},{172,31,255,255},eth2}). {{0,0,0,0},{255,255,255,255},eth0} which is exactly what you want. Of course, this need four calls to ets:prev/2, which (if time critical) could be avoided by using a flattened table (without holes) instead. In that case, only a single ets:prev/2 call (and nothing else from ets) would be necessary for all lookups. BTW, the example implementation I have given also returns (let's name the module 'iproute'). 1> R=iproute:new(). 28688 2> iproute:addroute(R,{0,0,0,0},{255,255,255,255},eth0). true 3> iproute:addroute(R,{192,168,0,0},{192,168,255,255},eth1). true 4> iproute:addroute(R,{172,16,0,0},{172,31,255,255},eth2). true 5> iproute:addroute(R,{192,168,1,0},{192,168,1,255},eth3). true 6> iproute:getroute(R,{239,0,0,1}). eth0 Jacob >> >> This can also be done without converting the IP adresses to binaries. >> The route specific data can also be put into the key to avoid an >> additional lookup as in >> >> E = ets:new(ip, [ordered_set]). >> ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}). >> ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}). >> ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}). >> >> {Low, High, If} = ets:prev(E, {{172,25,4,3},{256,256,256,256}, '_'}). >> -> {{172,16,0,0},{172,31,255,255},eth2} > > E = ets:new(ip, [ordered_set]), > ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}), > ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}), > ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}), > ets:insert(E, {{{192,168,1,0},{192,168,1,255},eth3},1}), > {Low, Hi, If} = ets:prev(E, {{192,168,10,11}, {256,256,256,256}, '_'}). > {{192,168,1,0},{192,168,1,255},eth3} > > As you see, lookup for 192.168.10.11 returns incorrect network 192.168.1.0/24, > not 192.168.0.0/16. Another example with the same setup: > > ets:prev(E, {{239,0,0,1}, {256,256,256,256}, '_'}). > {{192,168,1,0},{192,168,1,255},eth3} > > lookup for 239.0.0.1 returns 192.168.1.0/24, not 0.0.0.0/0 as it should. > > From sergej.jurecko@REDACTED Sun Nov 22 19:13:02 2015 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Sun, 22 Nov 2015 19:13:02 +0100 Subject: [erlang-questions] IP search optimization In-Reply-To: <5652033B.1030108@gmx.net> References: <35D58A8F-E63A-4666-B339-134A21EE9E6C@erlang.geek.nz> <56502864.9030008@gmx.net> <20151122171158.GB13944@staff.retn.net> <5652033B.1030108@gmx.net> Message-ID: <81F91847-862F-4C7B-8CD3-CC6D91B89DDA@gmail.com> Excuse me if this is a stupid comment. But why use ETS and cidr format at all if your goal is to find an IP that fits into one of your ranges. You can expand every range into {IpFrom,IpTo}. Sort them and create a binary tree by recursively half-ing it. Every element would be: {IPFrom, IPTo, LeftSubtree,RightSubtree}. Then it is a very simple matter of looking up your IP by traversing the tree. Sergej > On 22 Nov 2015, at 19:02, Jacob wrote: > > Hi Alexandre, > >> E = ets:new(ip, [ordered_set]), >> ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}), >> ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}), >> ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}), >> ets:insert(E, {{{192,168,1,0},{192,168,1,255},eth3},1}), >> {Low, Hi, If} = ets:prev(E, {{192,168,10,11}, {256,256,256,256}, '_'}). >> {{192,168,1,0},{192,168,1,255},eth3} >> >> As you see, lookup for 192.168.10.11 returns incorrect network > 192.168.1.0/24, >> not 192.168.0.0/16. Another example with the same setup: >> >> ets:prev(E, {{239,0,0,1}, {256,256,256,256}, '_'}). >> {{192,168,1,0},{192,168,1,255},eth3} >> >> lookup for 239.0.0.1 returns 192.168.1.0/24, not 0.0.0.0/0 as it should. >> > > That is why I wrote > > On 22.11.2015 18:11, Alexandre Snarskii wrote: >> On Sat, Nov 21, 2015 at 09:16:36AM +0100, Jacob wrote: >>> >>> In that case you can just use ets:prev to get the network, as in >>> >>> {Low, High} = ets:prev(E, {IP, <<255,255,255,255,1>>}). >>> > > That is crucial here: > >>> and a check whether IP <= High holds. > > Since {239,0,0,1} > {192,168,1,255} we are not done yet. > >>> The <<255,255,255,255,1>> is an >>> arbitrary value that just needs to be larger than any valid IP address, >>> to make lookups like for IP = <<192,168,0,0>> reliable. > > And further more: > >>> If you have >>> nested routes, you will have to call ets:prev({Low,High}) again until IP >>> <= High or $end_of_table is returned. > > Which tells how to get the correct result: > > 3> ets:prev(E, {{192,168,1,0},{192,168,1,255},eth3}). > {{192,168,0,0},{192,168,255,255},eth1} > 4> ets:prev(E, {{192,168,0,0},{192,168,255,255},eth1}). > {{172,16,0,0},{172,31,255,255},eth2} > 5> ets:prev(E, {{172,16,0,0},{172,31,255,255},eth2}). > {{0,0,0,0},{255,255,255,255},eth0} > > which is exactly what you want. > > Of course, this need four calls to ets:prev/2, which (if time critical) > could be avoided by using a flattened table (without holes) instead. In > that case, > only a single ets:prev/2 call (and nothing else from ets) would be > necessary for all lookups. > > BTW, the example implementation I have given also returns (let's name > the module 'iproute'). > > 1> R=iproute:new(). > 28688 > 2> iproute:addroute(R,{0,0,0,0},{255,255,255,255},eth0). > true > 3> iproute:addroute(R,{192,168,0,0},{192,168,255,255},eth1). > true > 4> iproute:addroute(R,{172,16,0,0},{172,31,255,255},eth2). > true > 5> iproute:addroute(R,{192,168,1,0},{192,168,1,255},eth3). > true > 6> iproute:getroute(R,{239,0,0,1}). > eth0 > > Jacob > > >>> >>> This can also be done without converting the IP adresses to binaries. >>> The route specific data can also be put into the key to avoid an >>> additional lookup as in >>> >>> E = ets:new(ip, [ordered_set]). >>> ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}). >>> ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}). >>> ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}). >>> >>> {Low, High, If} = ets:prev(E, {{172,25,4,3},{256,256,256,256}, '_'}). >>> -> {{172,16,0,0},{172,31,255,255},eth2} >> >> E = ets:new(ip, [ordered_set]), >> ets:insert(E, {{{0,0,0,0},{255,255,255,255},eth0},1}), >> ets:insert(E, {{{192,168,0,0},{192,168,255,255},eth1},1}), >> ets:insert(E, {{{172,16,0,0},{172,31,255,255},eth2},1}), >> ets:insert(E, {{{192,168,1,0},{192,168,1,255},eth3},1}), >> {Low, Hi, If} = ets:prev(E, {{192,168,10,11}, {256,256,256,256}, '_'}). >> {{192,168,1,0},{192,168,1,255},eth3} >> >> As you see, lookup for 192.168.10.11 returns incorrect network 192.168.1.0/24, >> not 192.168.0.0/16. Another example with the same setup: >> >> ets:prev(E, {{239,0,0,1}, {256,256,256,256}, '_'}). >> {{192,168,1,0},{192,168,1,255},eth3} >> >> lookup for 239.0.0.1 returns 192.168.1.0/24, not 0.0.0.0/0 as it should. >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Sun Nov 22 19:21:38 2015 From: zxq9@REDACTED (zxq9) Date: Mon, 23 Nov 2015 03:21:38 +0900 Subject: [erlang-questions] Erlang and Docker In-Reply-To: References: <1778791.TdEkoO8Abt@changa> Message-ID: <4134461.2t8HneJkov@changa> On 2015?11?22? ??? 11:27:47 Kevin Montuori wrote: > >>>>> "z" == zxq9 writes: > > z> I'm not saying these things don't work -- they do. But they have > z> never actually made deployment or maintenance any easier over the > z> long term -- instead merely move the complexity involved in those > z> tasks elsewhere for a while. > > Bringing up Docker certainly roused up some passions. I'm not sold on > containers as the solution to the world's ills but I'm also not willing > to eschew their use just because they're not always ideal. Right tool > for the job and all that. Well put. > Anyhow, I apologize for not being specific with the question I asked. I apologize for becoming passionate -- it drove me out on a tangent. Cathartic, but still a tangent. > I have a use case where containers are indicated and need a very small > distributed cache. Mnesia's my first choice but only if I don't have to > figure out the Erlang-in-Docker clustering myself; I was hoping someone > might have insight into (or a recipe for!) making that work. If not > there are other easily implementable solutions that I'll investigate. This is a curious phrase. You want "containers to have a cache"... ? Mnesia is *ideal* for creating caches. I actually don't really like DETS so much because it distracts from what Mnesia is really good at, and that is core cache with genuine db features. But... containers and in-memory caches with interesting features are usually viewed as orthogonal features. In the case of Erlang, they are really per-node concepts (unless you count DETS into that, then the discussion changes a bit, but is still manageable). Can you elaborate? I almost guarantee that someone will have a lot of interesting things to say about whatever it is you are dealing with. My suspicion is that the point at which the word "docker" entered the discussion is the point at which it became an X-Y problem. (And I went off on my tangent -- (>.<) dhoh. I appreciate your grace on this point.) -Craig From montuori@REDACTED Sun Nov 22 19:31:38 2015 From: montuori@REDACTED (Kevin Montuori) Date: Sun, 22 Nov 2015 12:31:38 -0600 Subject: [erlang-questions] Erlang and Docker In-Reply-To: <4134461.2t8HneJkov@changa> (zxq9@zxq9.com's message of "Mon, 23 Nov 2015 03:21:38 +0900") References: <1778791.TdEkoO8Abt@changa> <4134461.2t8HneJkov@changa> Message-ID: >>>>> "z" == zxq9 writes: >> I have a use case where containers are indicated and need a very small >> distributed cache. Mnesia's my first choice but only if I don't have to >> figure out the Erlang-in-Docker clustering myself; I was hoping someone >> might have insight into (or a recipe for!) making that work. If not >> there are other easily implementable solutions that I'll investigate. z> This is a curious phrase. You want "containers to have a cache"... ? z> Mnesia is *ideal* for creating caches. I actually don't really z> like DETS so much because it distracts from what Mnesia is really z> good at, and that is core cache with genuine db features. I should probably have phrased that as: I have an Erlang application where containers are indicated; that application requires a small cache. In a strictly OTP application deployment I would without doubt use Mnesia. But because the application has the Docker requirement things are a little more difficult. I agree 100% with Mnesia as distributed cache (and, honestly, for durable storage as well): it's treated me well over the years. k. -- Kevin Montuori montuori@REDACTED From on-your-mark@REDACTED Mon Nov 23 09:00:38 2015 From: on-your-mark@REDACTED (YuanZhiqian) Date: Mon, 23 Nov 2015 16:00:38 +0800 Subject: [erlang-questions] performing lists:keysearch on a list of records Message-ID: Hi guys, Glad to meet everyone here. I have a little question about the lists:keysearch function that I would like to search a record in a list by using one of its fields, because lists:keysearch is tuple-oriented and records is implemented internally as tuples, so I this the function should work as well to records, and it does as I tested in the shell. ( lists:keysearch(Key, 2, List), supposing the key is in the first field ) However I wonder if there's any side effects in this behavior, since I was taught that never access records' fields directly in the way of treating tuples. P.S. There's another question here, is there any difference between lists:keyfind/3 and lists:keysearch/3? I can tell any difference between them except for the return value's formats. MvhZhiqian -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Mon Nov 23 09:07:38 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Mon, 23 Nov 2015 09:07:38 +0100 Subject: [erlang-questions] performing lists:keysearch on a list of records In-Reply-To: References: Message-ID: lists:keyfind is faster and returns result directly instead of {value,Result}. For records, you should use: lists:keyfind(Value,#myrecord.myelement, List). Compiler will turn: #myrecord.myelement into a tuple position. Sergej On Mon, Nov 23, 2015 at 9:00 AM, YuanZhiqian wrote: > Hi guys, > > Glad to meet everyone here. > > I have a little question about the lists:keysearch function that I would > like to search a record in a list by using one of its fields, because > lists:keysearch is tuple-oriented and records is implemented internally as > tuples, so I this the function should work as well to records, and it does > as I tested in the shell. ( lists:keysearch(Key, 2, List), supposing the > key is in the first field ) > > However I wonder if there's any side effects in this behavior, since I > was taught that never access records' fields directly in the way of > treating tuples. > > P.S. There's another question here, is there any difference between > lists:keyfind/3 and lists:keysearch/3? I can tell any difference between > them except for the return value's formats. > > Mvh > Zhiqian > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Mon Nov 23 09:08:38 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Mon, 23 Nov 2015 10:08:38 +0200 Subject: [erlang-questions] performing lists:keysearch on a list of records In-Reply-To: References: Message-ID: Hello, You are on safe side with you approach. Just use #Name.Field The following expression returns the position of the specified field in the tuple representation of the record as it is defined by http://erlang.org/doc/reference_manual/records.html Best Regards, Dmitry > On Nov 23, 2015, at 10:00 AM, YuanZhiqian wrote: > > Hi guys, > > Glad to meet everyone here. > > I have a little question about the lists:keysearch function that I would like to search a record in a list by using one of its fields, because lists:keysearch is tuple-oriented and records is implemented internally as tuples, so I this the function should work as well to records, and it does as I tested in the shell. ( lists:keysearch(Key, 2, List), supposing the key is in the first field ) > > However I wonder if there's any side effects in this behavior, since I was taught that never access records' fields directly in the way of treating tuples. > > P.S. There's another question here, is there any difference between lists:keyfind/3 and lists:keysearch/3? I can tell any difference between them except for the return value's formats. > > Mvh > Zhiqian > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Mon Nov 23 09:58:46 2015 From: zxq9@REDACTED (zxq9) Date: Mon, 23 Nov 2015 17:58:46 +0900 Subject: [erlang-questions] performing lists:keysearch on a list of records In-Reply-To: References: Message-ID: <2894056.pIAOGp3kqM@changa> Hi Zhiqian, This question just came up the other day on SO: http://stackoverflow.com/questions/33846975/searching-through-lists-of-records-with-repeating-elements As usual, Steve was all over it, and actually provided this exact solution. On 2015?11?23? ??? 09:07:38 Sergej Jure?ko wrote: > lists:keyfind is faster and returns result directly instead of > {value,Result}. > > For records, you should use: lists:keyfind(Value,#myrecord.myelement, List). > > Compiler will turn: #myrecord.myelement into a tuple position. It seems like the "records -> goblins?" translation is being asked about a bit more than usual these days, so here is a way to check exactly what code results after precompilation by doing "erlc -E somemodule.erl". Here is an example module (based on Steve's SO answer): -module(recfind). -export([find_by_phone/2, find_by_mail/2]). -record(contact, {fname, lname, phone=[], mail=[], city=[], street=[]}). find_by_phone(Phone, AddressBook) -> find(Phone, #contact.phone, AddressBook). find_by_mail(Mail, AddressBook) -> find(Mail, #contact.mail, AddressBook). find(Value, Field, AddressBook) -> case lists:keyfind(Value, Field, AddressBook) of #contact{fname=Fname, lname=Lname} -> {Fname, Lname}; false -> {error, not_found} end. Here is the command to produce the source file after all source translations, but before compilation: ceverett@REDACTED:~/Code/erlang$ erlc -E recfind.erl And here is the resulting translated code (in a file called "recfind.E"): -file("recfind.erl", 1). find_by_phone(Phone, AddressBook) -> find(Phone, 4, AddressBook). find_by_mail(Mail, AddressBook) -> find(Mail, 5, AddressBook). find(Value, Field, AddressBook) -> case lists:keyfind(Value, Field, AddressBook) of {contact,Fname,Lname,_,_,_,_} -> {Fname,Lname}; false -> {error,not_found} end. module_info() -> erlang:get_module_info(recfind). module_info(X) -> erlang:get_module_info(recfind, X). We see that `find(Mail, #contact.mail, AddressBook)` has been translated to `find(Mail, 5, AddressBook)`. Sometimes this is interesting to do when a lot of preprocessing magic is involved in a source file and you want to know what the effect some construct or environment settings actually has in the resulting code (did those macros really fire?). Other interesting compiler options are `-P` and `-S`. The -E, -P, and -S switches for erlc are the same as the 'E', 'P', and 'S' options for compile:file/2 (http://www.erlang.org/doc/man/compile.html#file-2). -Craig From bchesneau@REDACTED Mon Nov 23 12:06:06 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 23 Nov 2015 11:06:06 +0000 Subject: [erlang-questions] where it's the best way to store a very big term object shared between processes In-Reply-To: References: <1445512774-410366-75.488472982304-29604@mail.muni.cz> Message-ID: On Thu, Oct 22, 2015 at 1:58 PM Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > A shot to try is to use a map instead. A trie should compress better on > paper, but I'm not sure the overhead in Erlang makes it competitive with > just shoving everything into a map (on 18.x), which uses a HAMT. > > I am not sure to understand the solution. Do you mean copying the map to different processes? What would be a good default number of processes then o start to copy the data? (/me is thinking N = number of cpus but...) Related to that I am wondering what `read_concurrency` means internally? - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Nov 23 12:21:58 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 23 Nov 2015 12:21:58 +0100 Subject: [erlang-questions] where it's the best way to store a very big term object shared between processes In-Reply-To: References: <1445512774-410366-75.488472982304-29604@mail.muni.cz> Message-ID: On Mon, Nov 23, 2015 at 12:06 PM, Benoit Chesneau wrote: > > I am not sure to understand the solution. Do you mean copying the map to > different processes? What would be a good default number of processes then > o start to copy the data? (/me is thinking N = number of cpus but...) > > maps() has no sharing, so in that case you need a single process to contain all of the data. Or you would have to split the data among several processes by level compressing the first levels of the radix tree. But this assumes you have roughly uniform distribution in your lookups. > Related to that I am wondering what `read_concurrency` means internally? > Mutexes around ETS are RX-mutexes. That is, they support either many readers or one (eXclusive) writer. But there are several ways to construct such an RX-mutex. In particular, you can make sure that if two or more cores want to read, they don't need to synchronize caches and be coherent. Such a "reader-optimized" mutex is selected when you enable read_concurrency. It makes writes more expensive in the sense you have to grab locks across all cores, but it makes the common read much faster. I think Rickard Green and/or Sverker Eriksson and/or Bj?rn-Egil Dahlberg knows more about these details if you are interested. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From on-your-mark@REDACTED Mon Nov 23 11:06:27 2015 From: on-your-mark@REDACTED (YuanZhiqian) Date: Mon, 23 Nov 2015 18:06:27 +0800 Subject: [erlang-questions] Update with complicated data structure Message-ID: Hi Guys, Sorry to bother you again... I am totally a green hand in Erlang :( Here is the problem I am now struggling with: in C&C++, we are always able to update a field in an object simply using obj.field1 = Value; But in Erlang, since its once-bound-never-change policy, people would have to find a way around to achieve the same goal, and things are getting worse when the data structure is a little more complicated, which is my case: I have a list of records, whose definition is as follows -record(company_info, { company_id, budget, consumption, compaign_ids }). What I want to do is to add a value to the consumption of the record whose company_id is given, and returning error message if there's no such record found. Written in C, that would be: ///////////////////////////////////In C&C++///////////CompanyInfo co_list[1000];/* initialized somewhere else ... */ for (size_t i = 0; i < 1000; ++i) { if (co_list[i].comany_id == co_id) { co_list[i].consumption += price; break; }} if (i == 1000) err_msg = "bla...";////////////////////////////////////////////////////// But in Erlang, I couldn't find a straightforward way to do this, my code is like this: ////////////////////////////In Erlang//////////////////////////%Co_list is the list of company_info records cal_win_notice({Co_id, Ca, Adgrp, Price, Cur}, #state{company_list = Co_list, campaign_list= Ca_list} = State) -> case lists:any(fun(#company_info{company_id = A}) -> A == Co_id end, Co_list) of true -> New_co_list = lists:map(fun(R) -> case R#company_info.company_id of Co_id -> R#company_info{consumption = R#company_info.consumption + Price}; _ -> R end end, Co_list), {ok, State#state{company_list = New_co_list}}; false -> {not_found, State} end. Well, as shown above, the codes in bold fonts are doing the same logic which in C can be done in just one expression. This will certainly make the code unnecessarily verbose when there are more similar operations to come. What should I do? I think this is due to Erlang's feature, but in case I am wrong and making a simple thing complicated ... I would appreciate so much with any advises. Best regardsZhiqian -------------- next part -------------- An HTML attachment was scrubbed... URL: From fernando.benavides@REDACTED Mon Nov 23 13:04:03 2015 From: fernando.benavides@REDACTED (Brujo Benavides) Date: Mon, 23 Nov 2015 09:04:03 -0300 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: Message-ID: <2F0552F7-680D-43C1-A250-999BFFAD83C9@inakanetworks.com> Hi Yuan, I have proposed two alternatives here . ?brujo1? assumes you don?t care about the order of the companies in your company_list. ?brujo2? is entirely compatible with your function. Cheers! > On Nov 23, 2015, at 07:06, YuanZhiqian wrote: > > Hi Guys, > > Sorry to bother you again... I am totally a green hand in Erlang :( > > Here is the problem I am now struggling with: in C&C++, we are always able to update a field in an object simply using obj.field1 = Value; But in Erlang, since its once-bound-never-change policy, people would have to find a way around to achieve the same goal, and things are getting worse when the data structure is a little more complicated, which is my case: > > I have a list of records, whose definition is as follows > > -record(company_info, { > company_id, > budget, > consumption, > compaign_ids > }). > > What I want to do is to add a value to the consumption of the record whose company_id is given, and returning error message if there's no such record found. Written in C, that would be: > > ///////////////////////////////////In C&C++/////////// > CompanyInfo co_list[1000]; > /* initialized somewhere else ... */ > > for (size_t i = 0; i < 1000; ++i) { > if (co_list[i].comany_id == co_id) { > co_list[i].consumption += price; > break; > } > } > > if (i == 1000) > err_msg = "bla..."; > ////////////////////////////////////////////////////// > > But in Erlang, I couldn't find a straightforward way to do this, my code is like this: > > ////////////////////////////In Erlang////////////////////////// > %Co_list is the list of company_info records > > cal_win_notice({Co_id, Ca, Adgrp, Price, Cur}, > #state{company_list = Co_list, campaign_list= Ca_list} = State) -> > case lists:any(fun(#company_info{company_id = A}) -> A == Co_id end, > Co_list) of > true -> > New_co_list = lists:map(fun(R) -> > case R#company_info.company_id of > Co_id -> > R#company_info{consumption = R#company_info.consumption + Price}; > _ -> > R > end > end, > Co_list), > {ok, State#state{company_list = New_co_list}}; > false -> > {not_found, State} > end. > > Well, as shown above, the codes in bold fonts are doing the same logic which in C can be done in just one expression. This will certainly make the code unnecessarily verbose when there are more similar operations to come. What should I do? I think this is due to Erlang's feature, but in case I am wrong and making a simple thing complicated ... > > I would appreciate so much with any advises. > > Best regards > Zhiqian > > > > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Nov 23 15:09:37 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 23 Nov 2015 15:09:37 +0100 Subject: [erlang-questions] Log SSL errors' originating IP addresses Message-ID: All, I'm using Cowboy with SSL, and I'm often seeing errors like these ones: SSL: hello: tls_handshake.erl:174:Fatal error: protocol version SSL: certify: ssl_alert.erl:93:Fatal error: illegal parameter SSL: hello: tls_handshake.erl:167:Fatal error: insufficient security And so on. To my understanding, these are pretty normal and result from a variety of reasons (attacks, bad clients using deprecated SSL protocols, etc). My question is, though: is there any way to log the IP addresses that originate these errors? Since these happen at Erlang level, they do not even hit my application and am unsure of what my best options are. Thank you for reading this. Best, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaiduanx@REDACTED Mon Nov 23 15:46:41 2015 From: kaiduanx@REDACTED (Kaiduan Xie) Date: Mon, 23 Nov 2015 09:46:41 -0500 Subject: [erlang-questions] Log SSL errors' originating IP addresses In-Reply-To: References: Message-ID: Looks like those errors happen at SSL handshake stage, you can log the IP address at your application. {ok, Socket} = ssl:transport_accept(Listen_socket) case ssl:ssl_accept(Socket) of ok-> .... {error, Reason} -> log the source IP of Socket here /Kaiduan On Mon, Nov 23, 2015 at 9:09 AM, Roberto Ostinelli wrote: > All, > I'm using Cowboy with SSL, and I'm often seeing errors like these ones: > > SSL: hello: tls_handshake.erl:174:Fatal error: protocol version > SSL: certify: ssl_alert.erl:93:Fatal error: illegal parameter > SSL: hello: tls_handshake.erl:167:Fatal error: insufficient security > > And so on. > > To my understanding, these are pretty normal and result from a variety of > reasons (attacks, bad clients using deprecated SSL protocols, etc). > > My question is, though: is there any way to log the IP addresses that > originate these errors? > Since these happen at Erlang level, they do not even hit my application and > am unsure of what my best options are. > > Thank you for reading this. > > Best, > r. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From rich.neswold@REDACTED Mon Nov 23 17:37:46 2015 From: rich.neswold@REDACTED (Rich Neswold) Date: Mon, 23 Nov 2015 10:37:46 -0600 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: Message-ID: On Mon, Nov 23, 2015 at 4:06 AM, YuanZhiqian wrote: > > -record(company_info, { > company_id, > budget, > consumption, > compaign_ids > }). > > cal_win_notice({Co_id, Ca, Adgrp, Price, Cur}, > #state{company_list = Co_list, campaign_list= Ca_list} = State) -> > case lists:any(fun(#company_info{company_id = A}) -> A == Co_id end, > Co_list) of > true -> > * New_co_list = lists:map(fun(R) -> * > * case R#company_info.company_id of* > * Co_id ->* > * R#company_info{consumption = > R#company_info.consumption + Price};* > * _ ->* > * R* > * end* > * end,* > * Co_list),* > * {ok, State#state{company_list = New_co_list}};* > false -> > {not_found, State} > end. > Something like this (not compiled or tested): cal_win_notice({Co_id, _, _, Price, _}, #state{company_list=Co_list}=State) -> case lists:mapfoldl(fun (#company_info{company_id=CID, consumption=C} = E, _) when CID == Co_id -> {E#company_info{consumption=C + price}, true}; (E, Acc) -> {E, Acc} end, false, Co_list) of {_, false} -> {not_found, State}; {NL, true} -> {ok, State#state{company_list=NL}} end. This only makes a single pass through the list. If you need to update more than one field, you do it in the same update. For instance, if you need to add $1000 to the budget, as well (ignoring whether it makes sense in this function) replace line 4 with: {E#company_info{consumption=C + price, budget=B + 1000}, true}; (you'll also have to pattern-match B with the budget field in line 2.) -- Rich -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Nov 23 17:57:36 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 23 Nov 2015 17:57:36 +0100 Subject: [erlang-questions] Log SSL errors' originating IP addresses In-Reply-To: References: Message-ID: Hi Kaiduan, Indeed, but I would most probably still need to catch errors (since these are raised at ssl level: https://github.com/otphub/ssl/blob/master/src/tls_handshake.erl#L174) However, the SSL handshake is called by Cowboy. I'm wondering if I can log this without patching it. Best, r. On Mon, Nov 23, 2015 at 3:46 PM, Kaiduan Xie wrote: > Looks like those errors happen at SSL handshake stage, you can log the > IP address at your application. > > {ok, Socket} = ssl:transport_accept(Listen_socket) > case ssl:ssl_accept(Socket) of > ok-> > .... > {error, Reason} -> > log the source IP of Socket here > > /Kaiduan > > On Mon, Nov 23, 2015 at 9:09 AM, Roberto Ostinelli > wrote: > > All, > > I'm using Cowboy with SSL, and I'm often seeing errors like these ones: > > > > SSL: hello: tls_handshake.erl:174:Fatal error: protocol version > > SSL: certify: ssl_alert.erl:93:Fatal error: illegal parameter > > SSL: hello: tls_handshake.erl:167:Fatal error: insufficient security > > > > And so on. > > > > To my understanding, these are pretty normal and result from a variety of > > reasons (attacks, bad clients using deprecated SSL protocols, etc). > > > > My question is, though: is there any way to log the IP addresses that > > originate these errors? > > Since these happen at Erlang level, they do not even hit my application > and > > am unsure of what my best options are. > > > > Thank you for reading this. > > > > Best, > > r. > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From denc716@REDACTED Mon Nov 23 18:55:05 2015 From: denc716@REDACTED (derek) Date: Mon, 23 Nov 2015 09:55:05 -0800 Subject: [erlang-questions] Erlang and Docker In-Reply-To: References: <1778791.TdEkoO8Abt@changa> <4134461.2t8HneJkov@changa> Message-ID: try the official image https://hub.docker.com/_/erlang/ for a good start for beginners, and there are other efforts like https://hub.docker.com/r/msaraiva/erlang/ making really minimal image, I think msaraiva is already running that in production ? docker run -it --rm erlang Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V7.1 (abort with ^G) 1> uptime(). 3 seconds ok 2> and for otp 18.1.4 https://github.com/docker-library/official-images/pull/1206 -------------- next part -------------- An HTML attachment was scrubbed... URL: From denc716@REDACTED Mon Nov 23 19:02:58 2015 From: denc716@REDACTED (derek) Date: Mon, 23 Nov 2015 10:02:58 -0800 Subject: [erlang-questions] where is os:getuid() ? Message-ID: Erlang/OTP 18 [erts-7.1] [source-2882b0c] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V7.1 (abort with ^G) 1> os:getpid(). "24801" 2> os:getuid(). ** exception error: undefined function os:getuid/0 3> I'm surprised such a fundamental api call is not there in erlang and has to be done in some c_driver like this one? wonder how many other POSIX system calls are not implemented in erlang standard library https://github.com/sergey-miryanov/erlang-setuid/blob/master/src/setuid.erl https://github.com/sergey-miryanov/erlang-setuid/blob/master/priv/setuid_drv.c -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Nov 23 20:09:54 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 23 Nov 2015 20:09:54 +0100 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: Message-ID: On Mon, Nov 23, 2015 at 7:02 PM, derek wrote: > wonder how many other POSIX system calls are not implemented in erlang > standard library Many! Most calls implemented is there due to a need by someone. This is why some of the system calls, however likely to be used, has not yet been implemented. Erlang doesn't in general provide a POSIX interface to the underlying Operating System, and many parts doesn't map well into the Erlang model. All hope is not lost however, since list_to_integer(string:strip(os:cmd("id -u"), right, $\n)). solves your problem. Granted, it performs worse than asking for the uid directly, but I've rarely seen a program who needed to manipulate uids otherwise, written in Erlang. May I ask what you are trying to do? -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Mon Nov 23 20:19:59 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Mon, 23 Nov 2015 20:19:59 +0100 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: Message-ID: Maybe such a thing can be implemented by the community and integrated in future releases? Now that we have an open issue tracker, could we put feature requests there and have people randomly work on them? Cheers, -- Pierre Fenoll On 23 November 2015 at 20:09, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > > On Mon, Nov 23, 2015 at 7:02 PM, derek wrote: > >> wonder how many other POSIX system calls are not implemented in erlang >> standard library > > > Many! Most calls implemented is there due to a need by someone. This is > why some of the system calls, however likely to be used, has not yet been > implemented. Erlang doesn't in general provide a POSIX interface to the > underlying Operating System, and many parts doesn't map well into the > Erlang model. > > All hope is not lost however, since > > list_to_integer(string:strip(os:cmd("id -u"), right, $\n)). > > solves your problem. Granted, it performs worse than asking for the uid > directly, but I've rarely seen a program who needed to manipulate uids > otherwise, written in Erlang. May I ask what you are trying to do? > > > -- > J. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam@REDACTED Mon Nov 23 20:34:20 2015 From: adam@REDACTED (Adam Cammack) Date: Mon, 23 Nov 2015 13:34:20 -0600 Subject: [erlang-questions] IP search optimization In-Reply-To: <20151122165345.GA13944@staff.retn.net> References: <20151119170510.GA72389@staff.retn.net> <20151120201804.GD8312@serenity> <20151122165345.GA13944@staff.retn.net> Message-ID: <20151123193420.GA2919@serenity> On Sun, Nov 22, 2015 at 07:53:45PM +0300, Alexandre Snarskii wrote: > This algorithm is good for your test set of non-overlapping networks, > but not in general. Let's assume we add entry {0, 0} (default route) > to your set and try to lookup some non-existant route. In case we look > for 244.42.192.0/20 - ets:prev returns 244.42.64.0/20 (last entry in > your set, which does not match), ets:prev for 244.42.64.0/20 returns > 244.42.48.0/20 (does not match again), then .32.0/20 and only after > ONE MILLION lookups you get matching 0.0.0.0/0 which is the right answer. > > With decrementing masklen you will get the same result in 32 lookups. My apologies, I forgot to mention successive calls to ets:prev/2 should done with the common prefix of the target IP and the non-matching block: {Common_Prefix, Common_Len + 1} common_prefix32(A, A) -> {A, 32}; common_prefix32(A, B) -> % Select the bits that are set in one, but not both Delta = (A bor B) bxor (A band B), % Most Significant Bit (first different) + 1 Last_Common = trunc(math:log2(Delta)) + 1, Prefix = (A bsr Last_Common) bsl Last_Common, {Prefix, 32 - Last_Common}. You could modify the algorithm to return all matching IP blocks by shifting out an additional bit of the match and continuing to search. In your case, the default rule would be found after 10 calls to ets:prev/2, being slightly slower in the worst case due to the extra math (same number of ETS operations). With nested blocks, the simplicity of naively searching for the mask is attractive. It would be interesting to see how the two methods scale with IPv6 addresses. I'll try to work up a benchmark later. -- Adam Cammack From jesper.louis.andersen@REDACTED Mon Nov 23 20:46:37 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 23 Nov 2015 20:46:37 +0100 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: Message-ID: On Mon, Nov 23, 2015 at 8:19 PM, Pierre Fenoll wrote: > Now that we have an open issue tracker, could we put feature requests > there and have people randomly work on them? > I think that is a good idea. getuid(2) and its cousin geteuid(2) never fails and returns quickly, so chances are they are safe to implement as a NIF, but we should definitely check if they may block or if they will take more than 1ms run-time to execute in the kernel. On OpenBSD, both calls are signal-handler-safe which is usually a good indication they are relatively safe to call. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Mon Nov 23 21:44:50 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Mon, 23 Nov 2015 22:44:50 +0200 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: Message-ID: <76791602-2F63-4541-B8D3-FEFCEB397DCD@gmail.com> Hello, One of the Erlang power is pattern match and recursions. I would look on the following suggestion for educational purposes: update_price([#company_info{company_id = Id, consumption = C}=Head | Tail], Id, Price) -> [Head#company_info{consumption = C + Price} | Tail]; update_price([Head | Tail], Id, Price) -> [Head | update_price(Tail, Id, Price)]. In real-life, you might look on some data structures to hold on your data set. Best Regards, Dmitry > On Nov 23, 2015, at 12:06 PM, YuanZhiqian wrote: > > Hi Guys, > > Sorry to bother you again... I am totally a green hand in Erlang :( > > Here is the problem I am now struggling with: in C&C++, we are always able to update a field in an object simply using obj.field1 = Value; But in Erlang, since its once-bound-never-change policy, people would have to find a way around to achieve the same goal, and things are getting worse when the data structure is a little more complicated, which is my case: > > I have a list of records, whose definition is as follows > > -record(company_info, { > company_id, > budget, > consumption, > compaign_ids > }). > > What I want to do is to add a value to the consumption of the record whose company_id is given, and returning error message if there's no such record found. Written in C, that would be: > > ///////////////////////////////////In C&C++/////////// > CompanyInfo co_list[1000]; > /* initialized somewhere else ... */ > > for (size_t i = 0; i < 1000; ++i) { > if (co_list[i].comany_id == co_id) { > co_list[i].consumption += price; > break; > } > } > > if (i == 1000) > err_msg = "bla..."; > ////////////////////////////////////////////////////// > > But in Erlang, I couldn't find a straightforward way to do this, my code is like this: > > ////////////////////////////In Erlang////////////////////////// > %Co_list is the list of company_info records > > cal_win_notice({Co_id, Ca, Adgrp, Price, Cur}, > #state{company_list = Co_list, campaign_list= Ca_list} = State) -> > case lists:any(fun(#company_info{company_id = A}) -> A == Co_id end, > Co_list) of > true -> > New_co_list = lists:map(fun(R) -> > case R#company_info.company_id of > Co_id -> > R#company_info{consumption = R#company_info.consumption + Price}; > _ -> > R > end > end, > Co_list), > {ok, State#state{company_list = New_co_list}}; > false -> > {not_found, State} > end. > > Well, as shown above, the codes in bold fonts are doing the same logic which in C can be done in just one expression. This will certainly make the code unnecessarily verbose when there are more similar operations to come. What should I do? I think this is due to Erlang's feature, but in case I am wrong and making a simple thing complicated ... > > I would appreciate so much with any advises. > > Best regards > Zhiqian > > > > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From s.kostyushkin@REDACTED Mon Nov 23 18:31:42 2015 From: s.kostyushkin@REDACTED (=?UTF-8?B?0KHQtdGA0LPRltC5INCa0L7RgdGC0Y7RiNC60ZbQvQ==?=) Date: Mon, 23 Nov 2015 19:31:42 +0200 Subject: [erlang-questions] SSL connection problem Message-ID: Do we have any official release with this patch? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Nov 23 22:22:55 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 23 Nov 2015 22:22:55 +0100 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: Message-ID: On Mon, Nov 23, 2015 at 11:06 AM, YuanZhiqian wrote: > Sorry to bother you again... I am totally a green hand in Erlang :( Suppose for the moment you are willing to use a map to store all your records. Then we can simplify the code quite a lot. List-representations are best used if you often process all of the the elements in the list, or you can use any element of the list, particularly the first one. If you are pin-pointing a specific entry, the map is a far better data structure. First a module definition: -module(z). -export([cal_win_notice/2]). Next, lets hack up a couple of records to match what is going on in your code: -record(state, { '...', companies, campaigns }). -record(company_info, { id, budget, consumption, campaigns }). The mapply/3 function here tries to apply F to the map value given by the key ID. If no such ID exists, it leaves the map alone. mapply(F, ID, Map) -> case maps:get(ID, Map, undefined) of undefined -> Map; T -> Map#{ ID := F(T) } end. Now, we can use this function. Make an F which does what we want. Then apply it. We can check if any change were made to the map by matching on the unchanged map. Otherwise, the map has been changed, and we can update the map with the new one. cal_win_notice({CoID, _Ca, _AdGrp, Price, _Cur}, #state { companies = Cos } = State) -> F = fun(#company_info{ consumption = Cons } = X) -> X#company_info{ consumption = Cons + Price } end, case mapply(F, CoID, Cos) of Cos -> {not_found, State}; %% Unchanged map Changed -> {ok, State#state { companies = Changed }} end. Want to make two changes? G = fun(#company_info { consumption = Cons } = X) -> X#company_info { consumption = Cons - Price } end, compose(F, G) -> fun(X) -> G(F(X)) end. then mapply(compose(F, G), CoID, Cos) ... -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandrashekhar.mullaparthi@REDACTED Tue Nov 24 00:04:05 2015 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Mon, 23 Nov 2015 23:04:05 +0000 Subject: [erlang-questions] spawn_link/2 behaviour In-Reply-To: References: Message-ID: Hi Avinash, See http://www.erlang.org/doc/getting_started/robustness.html#id69577 The default behaviour of a process that receives a normal exit is to ignore the signal. If you change the local_loop function as shown below, you will get the behaviour you are expecting. local_loop() -> receive stop -> io:format("stopped~n"), exit(stopped); X -> io:format("~p~n", [X]), local_loop() end. cheers, Chandru On 21 November 2015 at 11:49, Avinash Dhumane wrote: > Hello, > > I have following test program (on Windows): > > D:\Balin\Algo_v6\v0.5>type my_test.erl > -module(my_test). > -export([start/0]). > > start() -> > register(?MODULE, > spawn(fun() -> > init(), > local_loop() > end)). > > init() -> > spawn_link('feed@REDACTED', > fun() -> > register(?MODULE, self()), > remote_loop() > end), > > spawn('feed@REDACTED', > fun() -> > timer:sleep(timer:seconds(5)), % wait till the above process is > registered > ?MODULE ! hello, > ?MODULE ! world, > ?MODULE ! stop > end). > > remote_loop() -> > receive > X -> > {?MODULE, 'algo@REDACTED'} ! X, > remote_loop() > end. > > local_loop() -> > receive > stop -> > io:format("stopped~n"); > X -> > io:format("~p~n", [X]), > local_loop() > end. > > D:\Balin\Algo_v6\v0.5> > > Then, I start 2 nodes, as follows: > > D:\Balin\Algo_v6\v0.5>start erl -name algo@REDACTED > > D:\Balin\Algo_v6\v0.5>start erl -name feed@REDACTED > > On the 'algo@REDACTED' node, I run the program as follows: > > Eshell V7.0 (abort with ^G) > (algo@REDACTED)1> rpc:call('feed@REDACTED', erlang, whereis, > [my_test]). > undefined > (algo@REDACTED)2> my_test:start(). > true > (algo@REDACTED)3> hello > (algo@REDACTED)3> world > (algo@REDACTED)3> stopped > (algo@REDACTED)3> rpc:call('feed@REDACTED', erlang, whereis, > [my_test]). > <6755.43.0> > (algo@REDACTED)4> > > The messaging works as I expected. However, the process registered as > 'my_test' on the node 'feed@REDACTED' does not terminate. I expected it > to terminate since I started it using spawn_link/2 from the calling process > on 'algo@REDACTED' which terminates after receiving the 'stop' message. > > It keeps running, as is observed on the node 'feed@REDACTED': > > Eshell V7.0 (abort with ^G) > (feed@REDACTED)1> whereis(my_test). > <0.43.0> > (feed@REDACTED)2> > > What is wrong with the setup done in the test program above that the > linked process on another node does not terminate? > > Please advise. > > Thank you. > > Avinash > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.m.esposito@REDACTED Tue Nov 24 01:41:36 2015 From: tony.m.esposito@REDACTED (Anthony Esposito) Date: Tue, 24 Nov 2015 00:41:36 +0000 Subject: [erlang-questions] Simple SFTP Server Message-ID: Hi all, I'm trying to setup a simple Erlang SFTP server based on the documentation example from here: http://www.erlang.org/doc/apps/ssh/using_ssh.html#id59641 I'm struggling with it as my ssh client will only report "no hostkey alg" no matter how I configure the server. I've setup some public keys in a /tmp/ssh folder and tried the following code: ssh:start(). ssh:daemon(8989, [{system_dir, "/tmp/ssh"}, {user_dir, "/tmp/ssh"}, {subsystems, [ssh_sftpd:subsystem_spec([{cwd, "/tmp/ssh"}]) ]}]). I've both placed the keys in a .ssh folder and in the primary folder. I've experimented with different permissions and filenames but I'm not sure what the actual issue is. I've also tried forcing an algorithm with my client. sftp -oPort=8989 -o IdentityFile=/tmp/ssh/host_key -o 'HostKeyAlgorithms ssh-rsa,ssh-dss' localhost I'm trying to do this in userspace. If anyone has already figured this out I'd appreciate some tips to get me past this. Thank you, Tony -- ======================================== Anthony Esposito github/twitter: tesp0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.m.esposito@REDACTED Tue Nov 24 01:57:33 2015 From: tony.m.esposito@REDACTED (Anthony Esposito) Date: Tue, 24 Nov 2015 00:57:33 +0000 Subject: [erlang-questions] Simple SFTP Server In-Reply-To: References: Message-ID: Ok nevermind, of course after I finally ask the question I figure it out. I found some really specific instructions on how to setup the keys and it's working on my system now. FYI you have to follow these instructions here: http://www.erlang.org/doc/apps/ssh/using_ssh.html#id60714 Before moving onto anything else in the document. Thanks, Tony On Mon, Nov 23, 2015 at 6:41 PM Anthony Esposito wrote: > Hi all, > > I'm trying to setup a simple Erlang SFTP server based on the documentation > example from here: > http://www.erlang.org/doc/apps/ssh/using_ssh.html#id59641 > > I'm struggling with it as my ssh client will only report "no hostkey alg" > no matter how I configure the server. I've setup some public keys in a > /tmp/ssh folder and tried the following code: > > ssh:start(). > ssh:daemon(8989, [{system_dir, "/tmp/ssh"}, > {user_dir, "/tmp/ssh"}, > {subsystems, [ssh_sftpd:subsystem_spec([{cwd, "/tmp/ssh"}]) > ]}]). > > I've both placed the keys in a .ssh folder and in the primary folder. I've > experimented with different permissions and filenames but I'm not sure what > the actual issue is. > > I've also tried forcing an algorithm with my client. > > sftp -oPort=8989 -o IdentityFile=/tmp/ssh/host_key -o 'HostKeyAlgorithms > ssh-rsa,ssh-dss' localhost > > I'm trying to do this in userspace. > > If anyone has already figured this out I'd appreciate some tips to get me > past this. > > Thank you, > Tony > -- > ======================================== > Anthony Esposito > github/twitter: tesp0 > -- ======================================== Anthony Esposito github/twitter: tesp0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Nov 24 04:41:42 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 24 Nov 2015 16:41:42 +1300 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: Message-ID: <8A0D26AA-5D8E-45D3-B0C5-A74EFE10FD63@cs.otago.ac.nz> Erlang doesn't just run under Unix-family operating systems. If you're running under Cygwin, https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-common says how Windows identifiction <-> uid/gid work. I can't get a response from www.erlang.org at the moment, so I can't check this, but I thought Erlang could run directly under Windows without Cygwin. In that case, what should os:gete?[ug]id() do? Considering the hoops that Windows->uid/gid mapping has to (or may) jump through under Cygwin, it's not clear that their signal-safety under OpenBSD tells us much about Windows... From ok@REDACTED Tue Nov 24 05:34:05 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 24 Nov 2015 17:34:05 +1300 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: Message-ID: <437419A0-880C-4546-98FF-9DA5EFD639D0@cs.otago.ac.nz> On 23/11/2015, at 11:06 pm, YuanZhiqian wrote: > Hi Guys, > > Sorry to bother you again... I am totally a green hand in Erlang :( > > Here is the problem I am now struggling with: in C&C++, we are always able to update a field in an object simply using obj.field1 = Value; But in Erlang, since its once-bound-never-change policy, people would have to find a way around to achieve the same goal, Be VERY VERY CLEAR what "THE SAME GOAL" really means. It does *not* mean "fighting the language on the beaches, on the landing grounds, in the fields and the streets, (and) in the streets" and "never surrender"ing in your determination to *FORCE* the low level imperative style on your functional language. It means achieving the same APPLICATION-LEVEL GOAL (not code-level goal) by some possibly very different means. > and things are getting worse when the data structure is a little more complicated, which is my case: > > I have a list of records, whose definition is as follows > > -record(company_info, { > company_id, > budget, > consumption, > compaign_ids > }). > > What I want to do is to add a value to the consumption of the record whose company_id is given, and returning error message if there's no such record found. We stop right there. First, you are presuming a structure in which you don't KNOW whether a company-id is present or not. Why? Second, you are using a list, which is a very nice data structure, but the catalogue of things lists are good at does not include "searching". You might find gb_trees: more apt to your needs. case gb_trees:lookup(Company_Id, Companies) of none -> raise whatever exception you want ; {value,Company_Info=#company_info{budget=Old}} -> Updated = Company_Info#company_info{budget = Old + Delta}, gb_trees:update(Company_Id, Updated, Companies) end You should probably break this into two small functions: % (K, gbtree(K,V), (V -> V)) -> gbtree(K,V). update(Key, Tree, Fun) -> gb_trees:update(Key, Fun(gb_trees:get(Key, Tree)), Tree). increment_consumption(Info, Delta) -> Info#company_info(Info#company_info.consumption + Delta). ... update(Company_Id, Companies, fun (Info) -> increment_consumption(Info, Delta) end) ... > Written in C, that would be: > > ///////////////////////////////////In C&C++/////////// > CompanyInfo co_list[1000]; > /* initialized somewhere else ... */ > > for (size_t i = 0; i < 1000; ++i) { > if (co_list[i].comany_id == co_id) { > co_list[i].consumption += price; > break; > } > } > > if (i == 1000) > err_msg = "bla..."; > ////////////////////////////////////////////////////// Written in C that would be DISASTROUS if you ever had 1001 companies... (Where did this practice of using size_t for indices come from? That's what 'int' is for!) (a) You need to break your Erlang code into smaller, individually meaningful pieces. (b) You need to consider AND DOCUMENT why you want to use a data structure that takes linear time and turns over linear space for an update, when a data structure offering logarithmic time and space turnover is available. (c) List comprehensions are generally easier to read than calls to lists:map/2. (d) Avoid thinking in terms of Booleans; also, use pattern matching more. case [Info || Info = #company_info{company_id = Co_Id} <- Companies] of [Found] -> ... ; [] -> no match ; [_,_|_] -> multiple matches end Thinking in terms of lists:any made it impossible for you to notice duplicate entries; writing [Found] = [C || C = #company_info{company_id = Co_Id} <- Companies] makes it impossible NOT to notice. (e) Since the record is a 'company_info' record, what is the point of calling the key field 'company_id' rather than just 'id'? You didn't call 'budget' 'company_budget'. > > But in Erlang, I couldn't find a straightforward way to do this, my code is like this: > > ////////////////////////////In Erlang////////////////////////// > %Co_list is the list of company_info records > > cal_win_notice({Co_id, Ca, Adgrp, Price, Cur}, > #state{company_list = Co_list, campaign_list= Ca_list} = State) -> (f) Hvy abbrvtn mks ths hrd 2 ndrstnd. (g) Why do you have a mixed tuple that is not a record and is not separate arguments? > case lists:any(fun(#company_info{company_id = A}) -> A == Co_id end, > Co_list) of > true -> > New_co_list = lists:map(fun(R) -> > case R#company_info.company_id of > Co_id -> > R#company_info{consumption = R#company_info.consumption + Price}; > _ -> > R > end > end, > Co_list), > {ok, State#state{company_list = New_co_list}}; > false -> > {not_found, State} > end. No doubt you have a good reason for returning {not_found,State} instead of crashing (raising an exception), but it's probably worth documenting it. I say this because this is hard to compose: when you call your function you will have to handle two possible outcomes. Notice how my code got simpler when I switched from gb_trees:lookup/2 to gb_trees:get/2. > Well, as shown above, the codes in bold fonts are doing the same logic which in C can be done in just one expression. It wasn't one expression in C. It was 6 statements, including at least one capacity bug. You have to look at the *whole* thing, not just one tiny bit of it. > This will certainly make the code unnecessarily verbose when there are more similar operations to come. What should I do? I think this is due to Erlang's feature, but in case I am wrong and making a simple thing complicated ... Yes you are, by insisting that it be done the way you would do it in C. Make the code out of small composable pieces that are separately meaningful. From zxq9@REDACTED Tue Nov 24 05:39:28 2015 From: zxq9@REDACTED (zxq9) Date: Tue, 24 Nov 2015 13:39:28 +0900 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: <8A0D26AA-5D8E-45D3-B0C5-A74EFE10FD63@cs.otago.ac.nz> References: <8A0D26AA-5D8E-45D3-B0C5-A74EFE10FD63@cs.otago.ac.nz> Message-ID: <1908422.HnxqXQe9U2@burrito> On Tuesday 24 November 2015 16:41:42 Richard A. O'Keefe wrote: > Erlang doesn't just run under Unix-family operating > systems. If you're running under Cygwin, > https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-common > says how Windows identifiction <-> uid/gid work. > I can't get a response from www.erlang.org at the moment, > so I can't check this, but I thought Erlang could run > directly under Windows without Cygwin. In that case, > what should os:gete?[ug]id() do? > > Considering the hoops that Windows->uid/gid mapping has > to (or may) jump through under Cygwin, it's not clear > that their signal-safety under OpenBSD tells us much about > Windows... Indeed it does run under Windows, and does not require Cygwin. I read through the Windows build process a while back, and though a bit convoluted, iirc it is built with Visual Studio (and there were notes in there indicating that clean builds on mingw and VS was a goal). So... what should os:getuid/0 do? Does anyone see a downside to returning whatever is a native id relative to a given system in a tuple that indicates the nature of the system? In particular, whatever is consistent with os:type/0: -spec os:getuid() -> {System, Id} when System :: {unix, linux} | {unix, bsd} | {win, nt} % etc... Id :: unix_id() | win_id() % etc... Defining unix_id() may be too broad, and I have no idea how win_id() should be represented. The basic idea is letting different systems do their own things but tagging them in a way consistent with os:type/0. This could be a big win -- as long as the various types were well documented. Breaking too far out of a runtime cross-platform can get ugly pretty quickly, but I can imagine some uses for this. -Craig From nathaniel@REDACTED Mon Nov 23 22:41:05 2015 From: nathaniel@REDACTED (Nathaniel Waisbrot) Date: Mon, 23 Nov 2015 16:41:05 -0500 Subject: [erlang-questions] Erlang and Docker Message-ID: I'm using Docker with Erlang and have found it extremely useful. I think that there's some hostility towards Docker because much of the buzz around Docker seems to treat it as the perfect technology for solving all your problems. This is untrue, of course, but I think "stupid people like it" is a bad reason to reject something. So now I'll out myself as a stupid person... Here's an example of how I'm using Docker: https://gist.github.com/waisbrot/3415d53f7c0b66a36013 I use Kerl to install Erlang into a base image and then use that base image to build my applications. The applications are simply releases made with Rebar. There's a little Docker work to keep it from needing to download all dependencies every time, which is optional but nice. Inside the container, everything is normal Erlang world and there's no surprises. You could use the official image, but we wanted to customize our images and having a shared base image is valuable for saving downloading time and disk space. So that's the "how". As for the "why": - We started out in Node.js and Python. Containerization was helpful here, mostly to segregate Python2 and Python3 headaches (a bunch of tools that incompatibly expected "python" to be the right Python for them). - We were doing deployments via `git`, which was not great. Using Docker as a packaging system was nice because it's simpler to learn than RPMs and is uniform across all languages (don't have to learn wheels/eggs/gems/etc). - We didn't have any Ops staff to handle deployment and manage the machines - After the container infrastructure is set up, it's easy to introduce new languages and we were able to ease Erlang into the mix - Our development environment runs the same containers as our production environment, just using container-linking and running all on one machine - Or CI environment does the same thing. We regularly run our functional test suite against multiple copies of the production system simultaneously on a single machine (one run per pull-request), and it doesn't take any more effort than a single production deploy (which is also easy) My main complaint with Docker so far is that if I want my containers to launch with the system I need to set their restart policy to "always", but this causes it to restart crashed containers which interferes with Erlang's own restart system. Some day in the far future, we may reach a point where Erlang is the only language we're running (or at least the only one on a particular server) and then it might be attractive to un-containerize it. On the other hand, maybe not. Adding another layer of virtualization in the form of Docker has cost us very little so far. -------------- next part -------------- An HTML attachment was scrubbed... URL: From on-your-mark@REDACTED Tue Nov 24 07:51:03 2015 From: on-your-mark@REDACTED (YuanZhiqian) Date: Tue, 24 Nov 2015 14:51:03 +0800 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: <437419A0-880C-4546-98FF-9DA5EFD639D0@cs.otago.ac.nz> References: , <437419A0-880C-4546-98FF-9DA5EFD639D0@cs.otago.ac.nz> Message-ID: Hi Richard, Thanks for your elaborative comments. Some of the advices are helpful and I appreciate a lot for that; but some of them are not quite on the track ... For example, regarding the array size issue, you see, it's just a sample where my point is focused on the assignment expression, so I use an array instead of a hash table which I actually have used in my project, for that would make the code messed up with less important parts. I am certainly not a C language tard = = And please allow me to clarify my purpose of this question, this is not an erlang v.s. c&c++ debate, I am just facing an actual problem in real scenario, and need some practical advice. So it is not that I insist in making erlang behave the same as imperial language, but that it is a real requirement in the project that I'm trying to solve. As for the naming issue ... ok you are right, but does it really matter? At least not as important in my case ... And you are asking me why I am using size_t instead of int, I don't know if I am right, but size_t seems an intended type for array indices, since it is unassigned, and codes from google use size_t as well. Try to compile using int, but configure your error level to the most strict mode, turn on "warning as error" option, then you won't get the code compiled. So the point is, thanks again for your help, but unfortunately I am getting more confused with too much irrelevant comments. Best regardsZhiqian > Subject: Re: [erlang-questions] Update with complicated data structure > From: ok@REDACTED > Date: Tue, 24 Nov 2015 17:34:05 +1300 > CC: erlang-questions@REDACTED > To: on-your-mark@REDACTED > > > On 23/11/2015, at 11:06 pm, YuanZhiqian wrote: > > > Hi Guys, > > > > Sorry to bother you again... I am totally a green hand in Erlang :( > > > > Here is the problem I am now struggling with: in C&C++, we are always able to update a field in an object simply using obj.field1 = Value; But in Erlang, since its once-bound-never-change policy, people would have to find a way around to achieve the same goal, > > Be VERY VERY CLEAR what "THE SAME GOAL" really means. > > It does *not* mean "fighting the language on the beaches, > on the landing grounds, in the fields and the streets, > (and) in the streets" and "never surrender"ing in your > determination to *FORCE* the low level imperative style > on your functional language. > > It means achieving the same APPLICATION-LEVEL GOAL (not > code-level goal) by some possibly very different means. > > > and things are getting worse when the data structure is a little more complicated, which is my case: > > > > I have a list of records, whose definition is as follows > > > > -record(company_info, { > > company_id, > > budget, > > consumption, > > compaign_ids > > }). > > > > What I want to do is to add a value to the consumption of the record whose company_id is given, and returning error message if there's no such record found. > > We stop right there. > First, you are presuming a structure in which you don't > KNOW whether a company-id is present or not. > Why? > Second, you are using a list, which is a very nice data > structure, but the catalogue of things lists are good at > does not include "searching". You might find gb_trees: > more apt to your needs. > > case gb_trees:lookup(Company_Id, Companies) > of none -> raise whatever exception you want > ; {value,Company_Info=#company_info{budget=Old}} -> > Updated = > Company_Info#company_info{budget = Old + Delta}, > gb_trees:update(Company_Id, Updated, Companies) > end > > You should probably break this into two small functions: > > % (K, gbtree(K,V), (V -> V)) -> gbtree(K,V). > > update(Key, Tree, Fun) -> > gb_trees:update(Key, Fun(gb_trees:get(Key, Tree)), Tree). > > increment_consumption(Info, Delta) -> > Info#company_info(Info#company_info.consumption + Delta). > > ... update(Company_Id, Companies, fun (Info) -> > increment_consumption(Info, Delta) end) > ... > > > > Written in C, that would be: > > > > ///////////////////////////////////In C&C++/////////// > > CompanyInfo co_list[1000]; > > /* initialized somewhere else ... */ > > > > for (size_t i = 0; i < 1000; ++i) { > > if (co_list[i].comany_id == co_id) { > > co_list[i].consumption += price; > > break; > > } > > } > > > > if (i == 1000) > > err_msg = "bla..."; > > ////////////////////////////////////////////////////// > > Written in C that would be DISASTROUS if you ever had > 1001 companies... (Where did this practice of using > size_t for indices come from? That's what 'int' is for!) > > (a) You need to break your Erlang code into smaller, > individually meaningful pieces. > > (b) You need to consider AND DOCUMENT why you want to > use a data structure that takes linear time and > turns over linear space for an update, when a data > structure offering logarithmic time and space turnover > is available. > > (c) List comprehensions are generally easier to read than > calls to lists:map/2. > > (d) Avoid thinking in terms of Booleans; > also, use pattern matching more. > case [Info || Info = #company_info{company_id = Co_Id} <- Companies] > of [Found] -> ... > ; [] -> no match > ; [_,_|_] -> multiple matches > end > Thinking in terms of lists:any made it impossible for you > to notice duplicate entries; writing > [Found] = [C || C = #company_info{company_id = Co_Id} <- Companies] > makes it impossible NOT to notice. > > (e) Since the record is a 'company_info' record, what is the > point of calling the key field 'company_id' rather than > just 'id'? You didn't call 'budget' 'company_budget'. > > > > > > > But in Erlang, I couldn't find a straightforward way to do this, my code is like this: > > > > ////////////////////////////In Erlang////////////////////////// > > %Co_list is the list of company_info records > > > > cal_win_notice({Co_id, Ca, Adgrp, Price, Cur}, > > #state{company_list = Co_list, campaign_list= Ca_list} = State) -> > > (f) Hvy abbrvtn mks ths hrd 2 ndrstnd. > (g) Why do you have a mixed tuple that is not a record > and is not separate arguments? > > > case lists:any(fun(#company_info{company_id = A}) -> A == Co_id end, > > Co_list) of > > true -> > > New_co_list = lists:map(fun(R) -> > > case R#company_info.company_id of > > Co_id -> > > R#company_info{consumption = R#company_info.consumption + Price}; > > _ -> > > R > > end > > end, > > Co_list), > > {ok, State#state{company_list = New_co_list}}; > > false -> > > {not_found, State} > > end. > > No doubt you have a good reason for returning {not_found,State} > instead of crashing (raising an exception), but it's probably > worth documenting it. I say this because this is hard to > compose: when you call your function you will have to handle > two possible outcomes. Notice how my code got simpler when > I switched from gb_trees:lookup/2 to gb_trees:get/2. > > > Well, as shown above, the codes in bold fonts are doing the same logic which in C can be done in just one expression. > > It wasn't one expression in C. > It was 6 statements, including at least one capacity bug. > You have to look at the *whole* thing, not just one tiny > bit of it. > > > This will certainly make the code unnecessarily verbose when there are more similar operations to come. What should I do? I think this is due to Erlang's feature, but in case I am wrong and making a simple thing complicated ... > > Yes you are, > by insisting that it be done the way you would do it in C. > > Make the code out of small composable pieces that are > separately meaningful. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.abrahamsson@REDACTED Tue Nov 24 08:52:36 2015 From: daniel.abrahamsson@REDACTED (Daniel Abrahamsson) Date: Tue, 24 Nov 2015 08:52:36 +0100 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: <1908422.HnxqXQe9U2@burrito> References: <8A0D26AA-5D8E-45D3-B0C5-A74EFE10FD63@cs.otago.ac.nz> <1908422.HnxqXQe9U2@burrito> Message-ID: Rather than to try to find a generic interface for something that is inherently OS specific, perhaps it would be better to create a "posix" module that exports some common posix operations. That way it is obvious to whoever use it that you can't expect it to work on non-posix OSes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From on-your-mark@REDACTED Tue Nov 24 08:53:27 2015 From: on-your-mark@REDACTED (YuanZhiqian) Date: Tue, 24 Nov 2015 15:53:27 +0800 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: <2F0552F7-680D-43C1-A250-999BFFAD83C9@inakanetworks.com> , Message-ID: Hi Brujo, Thanks a lot. I think your approach is a best practice for these common data access in erlang. I am reading your codes to digest more details :) Best regardsZhiqian From: fernando.benavides@REDACTED Subject: Re: [erlang-questions] Update with complicated data structure Date: Mon, 23 Nov 2015 10:12:43 -0300 To: on-your-mark@REDACTED Hi Yuan, First of all, I usually use maps these days, not records. But, in any case, my usual approach is to hide data structures in their own modules, like the ones in this lsl folder (I?m talking about lsl_matches, lsl_players and lsl_sessions). Each one of them provides an opaque type to represent the entities they manage, along with functions to create/update/etc? the entities. If multiple simultaneous modifications are required by the system, I usually add an update function like: -spec update(entity(), updates()) -> entity(). update(Entity, Updates) -> Entity1 = maps:merge(Setting, Updates), Entity1#{updated_at := calendar:universal_time()}. That one works just because both entity() and updates() are maps. Sometimes (most of the times) update functions are more complex than that. In any case, business logic related to the entities is usually placed on the entity_repo modules (you can also find examples of them on the aforementioned folder). Hope this helps :) PS: You should use reply-all, so that your emails reach the erlang-questions mailing list as well :) On Nov 23, 2015, at 10:01, YuanZhiqian wrote:Hi Brujo, Thank you very much, that's a much better solution :) But still, I wonder how you and other guys normally deal with heavy works of states maintaining, since every time I want to update the system state, I would have to assign a new variable and pass it everywhere with function calls to maintain the records. I have checked out ETS (erlang term storage), it is better than records, since I don't need to pass values on every function call, you can access any table simply by using an ID or Name, but it is still not quite handy to update. Do we have to face the trouble with CRUD on data structures, or is it that I am not using the correct design principle with Erlang? P.S. to make this question less abstract, let me illustrate more with my scenario: In a function call, I may be checking some fields in a "obj" -- which would be a record in Erlang, and modify several other fields of it, so that means I need to assign a lot of temporary variables and return the record with new fields altogether. And this can be really complicated when the record is nested in a list; it is also painful to name a new variable when a field is changed a lot of times. Best regardsZhiqian Subject: Re: [erlang-questions] Update with complicated data structure From: fernando.benavides@REDACTED Date: Mon, 23 Nov 2015 09:04:03 -0300 CC: erlang-questions@REDACTED To: on-your-mark@REDACTED Hi Yuan, I have proposed two alternatives here. ?brujo1? assumes you don?t care about the order of the companies in your company_list. ?brujo2? is entirely compatible with your function. Cheers! On Nov 23, 2015, at 07:06, YuanZhiqian wrote:Hi Guys, Sorry to bother you again... I am totally a green hand in Erlang :( Here is the problem I am now struggling with: in C&C++, we are always able to update a field in an object simply using obj.field1 = Value; But in Erlang, since its once-bound-never-change policy, people would have to find a way around to achieve the same goal, and things are getting worse when the data structure is a little more complicated, which is my case: I have a list of records, whose definition is as follows -record(company_info, { company_id, budget, consumption, compaign_ids }). What I want to do is to add a value to the consumption of the record whose company_id is given, and returning error message if there's no such record found. Written in C, that would be: ///////////////////////////////////In C&C++///////////CompanyInfo co_list[1000];/* initialized somewhere else ... */ for (size_t i = 0; i < 1000; ++i) { if (co_list[i].comany_id == co_id) { co_list[i].consumption += price; break; }} if (i == 1000) err_msg = "bla...";////////////////////////////////////////////////////// But in Erlang, I couldn't find a straightforward way to do this, my code is like this: ////////////////////////////In Erlang//////////////////////////%Co_list is the list of company_info records cal_win_notice({Co_id, Ca, Adgrp, Price, Cur}, #state{company_list = Co_list, campaign_list= Ca_list} = State) -> case lists:any(fun(#company_info{company_id = A}) -> A == Co_id end, Co_list) of true -> New_co_list = lists:map(fun(R) -> case R#company_info.company_id of Co_id -> R#company_info{consumption = R#company_info.consumption + Price}; _ -> R end end, Co_list), {ok, State#state{company_list = New_co_list}}; false -> {not_found, State} end. Well, as shown above, the codes in bold fonts are doing the same logic which in C can be done in just one expression. This will certainly make the code unnecessarily verbose when there are more similar operations to come. What should I do? I think this is due to Erlang's feature, but in case I am wrong and making a simple thing complicated ... I would appreciate so much with any advises. Best regardsZhiqian _______________________________________________erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From on-your-mark@REDACTED Tue Nov 24 09:11:36 2015 From: on-your-mark@REDACTED (YuanZhiqian) Date: Tue, 24 Nov 2015 16:11:36 +0800 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: , Message-ID: Hi Jesper, Thanks a lot! I'll dig into it. Best regardsZhiqian From: jesper.louis.andersen@REDACTED Date: Mon, 23 Nov 2015 22:22:55 +0100 Subject: Re: [erlang-questions] Update with complicated data structure To: on-your-mark@REDACTED CC: erlang-questions@REDACTED On Mon, Nov 23, 2015 at 11:06 AM, YuanZhiqian wrote: Sorry to bother you again... I am totally a green hand in Erlang :( Suppose for the moment you are willing to use a map to store all your records. Then we can simplify the code quite a lot. List-representations are best used if you often process all of the the elements in the list, or you can use any element of the list, particularly the first one. If you are pin-pointing a specific entry, the map is a far better data structure. First a module definition: -module(z).-export([cal_win_notice/2]). Next, lets hack up a couple of records to match what is going on in your code: -record(state, { '...', companies, campaigns}). -record(company_info, { id, budget, consumption, campaigns}). The mapply/3 function here tries to apply F to the map value given by the key ID. If no such ID exists, it leaves the map alone. mapply(F, ID, Map) -> case maps:get(ID, Map, undefined) of undefined -> Map; T -> Map#{ ID := F(T) } end. Now, we can use this function. Make an F which does what we want. Then apply it. We can check if any change were made to the map by matching on the unchanged map. Otherwise, the map has been changed, and we can update the map with the new one. cal_win_notice({CoID, _Ca, _AdGrp, Price, _Cur}, #state { companies = Cos } = State) -> F = fun(#company_info{ consumption = Cons } = X) -> X#company_info{ consumption = Cons + Price } end, case mapply(F, CoID, Cos) of Cos -> {not_found, State}; %% Unchanged map Changed -> {ok, State#state { companies = Changed }} end. Want to make two changes? G = fun(#company_info { consumption = Cons } = X) -> X#company_info { consumption = Cons - Price } end, compose(F, G) -> fun(X) -> G(F(X)) end. then mapply(compose(F, G), CoID, Cos) ... -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Nov 24 09:26:19 2015 From: zxq9@REDACTED (zxq9) Date: Tue, 24 Nov 2015 17:26:19 +0900 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: <1908422.HnxqXQe9U2@burrito> Message-ID: <1959946.V3vEkza5Ge@burrito> On Tuesday 24 November 2015 08:52:36 Daniel Abrahamsson wrote: > Rather than to try to find a generic interface for something that is > inherently OS specific, perhaps it would be better to create a "posix" > module that exports some common posix operations. That way it is obvious to > whoever use it that you can't expect it to work on non-posix OSes. That is certainly one possibility. Which feels more inconvenient? do_some_os_thing() -> perform_os_action(os:type()) perform_os_action({unix, _}) -> {ok, NodeUid} = posix:getuid(), do_action(posix, NodeUid); perform_os_action({win, _}) -> {ok, NodeUid} = windows:getuid(), do_action(win, NodeUid); % ... do_action(posix, NodeUid) -> % stuff...; do_action(win, NodeUid) -> % slightly different stuff...; % ... OR do_some_os_thing() -> perform_os_action(os:getuid()) perform_os_action({{unix, _}, NodeUid}) -> % stuff... ; perform_os_action({{win, _}, NodeUid}) -> % slightly different stuff...; % ... OR do_some_os_thing() -> % Inventing an os:get_feature_module/1 here... OSType = os:type(), OSMod = os:get_feature_module(OSType) {ok, NodeUid} = OSMod:getuid(), perform_os_action(OSType, NodeUid) %% etc... I can't really see a clear balance of pros and cons either way -- the difference will *probably* have to be tackled head-on in the related code eventually, at least if that code is intended to work across platforms. I'm not sure that an OS-specific module will do much to make the calling code less annoying to write -- though it may make building ERTS against a given platform easier (you only need the module relevant to the target platform -- but I think this has to happen in any case underneath?). What I do at the moment is match on the result of os:type/0, and execute an os-specific script or command (shell, batchfile, some natively built code, whatever). As you say, one real benefit of making a 'posix', 'windows', etc. modules would be that determining exactly what features are supported on a given platform becomes as simple as checking the docs for that platform's module. That's better than having a generic interface that only works the same way on most but not all platforms. (I *really* hate hunting through docs for caveats like "super_important_function/3 only works on platform X; does nothing and returns 'ok' on Y"). On the other hand, checking for API compatibility would then require looking through each platform module to make sure you didn't miss anything. :-/ Any thoughts? I write a fair amount of cross-platform code in Erlang these days. This sort of thing is so annoying that I will do cheetah flips to avoid the weird places where os differences start mattering (if the need is non-trivial I'll write entirely separate native programs I can call from Erlang). Fortunately cases where this can't be avoided are rare. -Craig From daniel.abrahamsson@REDACTED Tue Nov 24 11:50:51 2015 From: daniel.abrahamsson@REDACTED (Daniel Abrahamsson) Date: Tue, 24 Nov 2015 11:50:51 +0100 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: <1959946.V3vEkza5Ge@burrito> References: <1908422.HnxqXQe9U2@burrito> <1959946.V3vEkza5Ge@burrito> Message-ID: With the risk of sidetracking from the original question... At a theoretical level, I prefer separating functions implementing policies, and functions implementing mechanisms. "Mechanism" functions do one specific thing. There are no exceptions or corner cases. "Policy" functions, deal with the exceptions. They implement a policy, i.e. they make a specific decision on how to handle a certain case. As an example, "platform A has no user concept, so I will always return uid 0, which is root". Another policy function could implement the policy "platform A has no user concept, so I will throw an exception". When you confuse, or merge, the two, you limit the opportunity to implement other policies. If you have the "mechanism" functions, you can implement whatever policy suits you. If you only have the policy function, the corner cases are going to bite you somewhere down the road. With this reasoning, having a posix module makes sense, because that is a "mechanism" thing. If I want to implement a generic function that works similar on both Windows and posix, I can put that in a separate module, which describes the chosen policy (i.e. the trade-offs made). getuid is probably not a good example here, because it is likely to make sense on most, if not all, platforms, but I hope you get the idea. I hope I didn't side-track the discussion too much. -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Tue Nov 24 19:18:56 2015 From: schneider@REDACTED (Frans Schneider) Date: Tue, 24 Nov 2015 19:18:56 +0100 Subject: [erlang-questions] Would like some feedback Message-ID: <5654AA10.3040007@xs4all.nl> Dear list, I've been learning Erlang in my spare-time for some time now and would love some feedback on a project I recently started. Since I have no coworkers or other people to get feedback from, maybe somebody on this list could have a quick look. The things I find particular difficult is defining a proper API, to decide what goes into a new process or is done in the calling process and dealing with errors. I have no formal training as a programmer which may be the reason I have trouble to get these things right. The project I took [1] was to implement the Axolotl ratcheting protocol from Open Whisper Systems [2]. The protocol [3] itself didn't look very difficult, but since there are no proper specs for the total system, I used the libaxolotl-java implementation as a starting pont [4]. Reading the Java code probably was the hardest part of the whole project. I think I covered most of the functionality of the libaxolotl-java library. I haven't verified interoperability between my implementation and libaxolotl-java since that would require me to start coding in Java which is not something I am looking forward to. Comparing my Erlang code with the original Java code, I needed far less lines of code and, I think, the code is much easier to understand. Also, building and deploying the code seems much simpler. So, any comments are most welcome. Thanks, Frans [1] https://github.com/schnef/axolotl [2] https://www.whispersystems.org/ [3] https://github.com/trevp/axolotl/wiki [4] https://github.com/WhisperSystems/libaxolotl-java Nice implementation https://github.com/rxcomm/pyaxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Tue Nov 24 19:47:06 2015 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Tue, 24 Nov 2015 19:47:06 +0100 Subject: [erlang-questions] Would like some feedback In-Reply-To: <5654AA10.3040007@xs4all.nl> References: <5654AA10.3040007@xs4all.nl> Message-ID: <907DB1FC-4B66-4992-9D67-FF48C0741D01@gmail.com> What I would suggest is separating the axolotl bits from mnesia/gen_fsm bits. Your library needlessly imposes an architecture on the user. Sergej > On 24 Nov 2015, at 19:18, Frans Schneider wrote: > > Dear list, > > I've been learning Erlang in my spare-time for some time now and would love some feedback on a project I recently started. Since I have no coworkers or other people to get feedback from, maybe somebody on this list could have a quick look. > The things I find particular difficult is defining a proper API, to decide what goes into a new process or is done in the calling process and dealing with errors. I have no formal training as a programmer which may be the reason I have trouble to get these things right. > > The project I took [1] was to implement the Axolotl ratcheting protocol from Open Whisper Systems [2]. The protocol [3] itself didn't look very difficult, but since there are no proper specs for the total system, I used the libaxolotl-java implementation as a starting pont [4]. Reading the Java code probably was the hardest part of the whole project. > > I think I covered most of the functionality of the libaxolotl-java library. I haven't verified interoperability between my implementation and libaxolotl-java since that would require me to start coding in Java which is not something I am looking forward to. > Comparing my Erlang code with the original Java code, I needed far less lines of code and, I think, the code is much easier to understand. Also, building and deploying the code seems much simpler. > > So, any comments are most welcome. > > Thanks, > > Frans > > [1] https://github.com/schnef/axolotl > [2] https://www.whispersystems.org/ > [3] https://github.com/trevp/axolotl/wiki > [4] https://github.com/WhisperSystems/libaxolotl-java > Nice implementation https://github.com/rxcomm/pyaxo > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From denc716@REDACTED Tue Nov 24 20:08:57 2015 From: denc716@REDACTED (derek) Date: Tue, 24 Nov 2015 11:08:57 -0800 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: <1908422.HnxqXQe9U2@burrito> <1959946.V3vEkza5Ge@burrito> Message-ID: On Mon, Nov 23, 2015 at 11:09 AM, Jesper Louis Andersen wrote: > All hope is not lost however, since > > list_to_integer(string:strip(os:cmd("id -u"), right, $\n)). > > solves your problem. Granted, it performs worse than asking for the uid directly, but I've rarely seen a program who needed to manipulate uids otherwise, written in Erlang. May I ask what you are trying to do? in rebar managed code, my building environment has different part using different users, some file owner changes are due to `user namespace`, run root with a normal user id, and need constantly fix uid/gid by chown , or need to detect what is current user, trying to implement in pure Erlang, but it seems Erlang doesn't have os:getuid/0, neither os:chown/3. Workaround is to do that in shell / python with Erlang os:cmd/1. 2> os:chown("abc", 1000, 100). ** exception error: undefined function os:chown/3 In this year Erlang factory they talked about to extend usage scenario of Erlang, but this lack of POSIX interfaces might be a large roadblock, continue Erlang's strength in long time running Internet services, but for small tasks, short lived programs / daily useful utilities, so far we only know erlc, rebar, ... and what else? mostly are better in Shell / Python, http://www.erlang-factory.com/sfbay2015/mike-williams If Consider to add os:getuid/0 to Erlang, why not adopt a similar strategy like Python os module, make it available for Unix only, no need to care windows, that platform lacks a lot of POSIX calls, https://docs.python.org/3/library/os.html On Tue, Nov 24, 2015 at 2:50 AM, Daniel Abrahamsson wrote: > With the risk of sidetracking from the original question... > > At a theoretical level, I prefer separating functions implementing policies, > and functions implementing mechanisms. "Mechanism" functions do one specific > thing. There are no exceptions or corner cases. "Policy" functions, deal > with the exceptions. They implement a policy, i.e. they make a specific > decision on how to handle a certain case. As an example, "platform A has no > user concept, so I will always return uid 0, which is root". Another policy > function could implement the policy "platform A has no user concept, so I > will throw an exception". When you confuse, or merge, the two, you limit the > opportunity to implement other policies. If you have the "mechanism" > functions, you can implement whatever policy suits you. If you only have the > policy function, the corner cases are going to bite you somewhere down the > road. From chandrashekhar.mullaparthi@REDACTED Wed Nov 25 00:55:33 2015 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Tue, 24 Nov 2015 23:55:33 +0000 Subject: [erlang-questions] Would like some feedback In-Reply-To: <5654AA10.3040007@xs4all.nl> References: <5654AA10.3040007@xs4all.nl> Message-ID: Hi Frans, Here are a few comments with the caveat that I haven't spent any time understanding the protocol. So the comments are more about implementation style... - Firstly, its quite impressive you've managed to write this in quite an idiomatic style for your hobby project, so well done! - You might want to consider naming all your modules with the 'axolotl_' prefix to minimise the possibility of module name clashes when others use your code - It would be a good idea to put all your DB access operations in a module such as axolotl_db.erl so that if you ever wanted to changed your database backend, the changes are limited to one module. It might not be that important for this particular application, but as rule of thumb it will serve you well. - You raise a good question about "... decide what goes into a new process or is done in the calling process...". I'll try and clarify this with an example. You have implemented identities:check_registration/2 as a gen_server:call, but the code which handles this message in the handle_call function is just reading from mnesia and returning a result. This need not be a gen_server:call, it can be written as follows. check(Registration_id, Dhi_pub) -> case mnesia:dirty_read(identity, Dhi_pub) of [#identity{registration_id = Registration_id}] -> true; _ -> false end. This is because there is no value in putting it through the gen_server. It still makes sense for the code to reside in this module as it logically belongs here. - Typically with Erlang applications, your mnesia initialisation code should be separate from your normal running code. It is okay to expect the user to run a one off installation function to create the required tables. Also, it is not a good idea to delete schema and recreate it because it becomes hard to use this as part of a node with other applications which have their own mnesia tables. - The code in prekeys.erl is a bit hard to read. You might want to move the code from each of the handle_call function clauses into separate functions which are invoked. This is probably partly due to my ignorance of the protocol, but the code could be made to look cleaner. - I couldn't see any test code in the modules, how are you testing this? I hope this is of some help. cheers, Chandru On 24 November 2015 at 18:18, Frans Schneider wrote: > Dear list, > > I've been learning Erlang in my spare-time for some time now and would > love some feedback on a project I recently started. Since I have no > coworkers or other people to get feedback from, maybe somebody on this list > could have a quick look. > The things I find particular difficult is defining a proper API, to decide > what goes into a new process or is done in the calling process and dealing > with errors. I have no formal training as a programmer which may be the > reason I have trouble to get these things right. > > The project I took [1] was to implement the Axolotl ratcheting protocol > from Open Whisper Systems [2]. The protocol [3] itself didn't look very > difficult, but since there are no proper specs for the total system, I used > the libaxolotl-java implementation as a starting pont [4]. Reading the Java > code probably was the hardest part of the whole project. > > I think I covered most of the functionality of the libaxolotl-java > library. I haven't verified interoperability between my implementation and > libaxolotl-java since that would require me to start coding in Java which > is not something I am looking forward to. > Comparing my Erlang code with the original Java code, I needed far less > lines of code and, I think, the code is much easier to understand. Also, > building and deploying the code seems much simpler. > > So, any comments are most welcome. > > Thanks, > > Frans > > [1] https://github.com/schnef/axolotl > [2] https://www.whispersystems.org/ > [3] https://github.com/trevp/axolotl/wiki > [4] https://github.com/WhisperSystems/libaxolotl-java > Nice implementation https://github.com/rxcomm/pyaxo > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Nov 25 05:13:23 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 25 Nov 2015 17:13:23 +1300 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: , <437419A0-880C-4546-98FF-9DA5EFD639D0@cs.otago.ac.nz> Message-ID: On 24/11/2015, at 7:51 pm, YuanZhiqian wrote: > Hi Richard, > > Thanks for your elaborative comments. Some of the advices are helpful and I appreciate a lot for that; but some of them are not quite on the track ... Well, if I'm not _shown_ the track it's hard to stay on it. > > For example, regarding the array size issue, you see, it's just a sample where my point is focused on the assignment expression, so I use an array instead of a hash table which I actually have used in my project, for that would make the code messed up with less important parts. I am certainly not a C language tard = = "Tard"? What does that mean? https://en.wikipedia.org/wiki/Tardigrade ? > > And please allow me to clarify my purpose of this question, this is not an erlang v.s. c&c++ debate, That was understood; no clarification needed. > I am just facing an actual problem in real scenario, and need some practical advice. While it's not a C vs Erlang debate, it *is* a matter of trying to do things the C way in an Erlang program. > > So it is not that I insist in making erlang behave the same as imperial language, but that it is a real requirement in the project that I'm trying to solve. I doubt that very much. Requirements talk about what a program does, as seen from outside. How the collection of companies is held, whether as a list, as a gb tree, as a map, as an ETS table, or even in the process dictionary (for safety, that had better be the process dictionary of a separate process) should be entirely up to you. I have serious trouble believing in requirements that say "You must use a data structure that makes search and update take a long time." Your requirements may well say "Update the consumption of company X by D" but of course you haven't shown us the requirements. One thing may have struck you already. If you are using a relational data base, changing one field of one record at a time is a good way to run quite slowly. One way to get good performance out of RDBs is *batching*. Now if you kept the list of N companies *sorted* on increasing company_id, and if you had a *batch* of M companies to be updated, also in a list sorted on increasing company_id, then you can *merge* those two lists and produce a new list in O(N+M) time. (This is what old-timers used to do with tape drives before even CODASYL data bases.) But if you do it one company at a time, it's going to take O(N*M) time. If you use a gb_tree, O(lg(N)*M). So there is a really serious question about whether you really *HAVE* to update one field of one record, or whether you have a *batch* of fields, and possibly multiple updates. > > As for the naming issue ... ok you are right, but does it really matter? Yes, it does. > At least not as important in my case ... > > And you are asking me why I am using size_t instead of int, I don't know if I am right, but size_t seems an intended type for array indices, (a) It cannot possibly be. C existed long before size_t did. (b) The name *tells* you what it's meant for: it is meant for the *sizes* of objects. (c) Kernighan & Ritchie, "The C programming language", 2nd edition consistently uses int for array indexing. But what do they know? They only invented the language. > since it is unassigned, I think you mean 'unsigned', and that's actually bad. > and codes from google use size_t as well. Google invented Go. They did not invent C and are not authorities on how it was meant to be used. > Try to compile using int, but configure your error level to the most strict mode, turn on "warning as error" option, then you won't get the code compiled. I have no idea what you are talking about. Code that indexes arrays using int gets through gcc, clang, lint, and splint at the highest levels with no warnings. Good C compilers warn about using *char* as an array index, because the programmer has no way of telling beforehand whether char is signed or unsigned, so a[(char)255] might or might not involve a negative subscript. But invalidating int would mean breaking almost all the C code ever written for Unix. I just checked the source code for (original) vi, and yep, it uses int subscripts, including "tp[-1]", which is not actually an unusual thing to do in C (tp being a pointer after the beginning of an array), and no usable C compiler is going to treat that as an error. > So the point is, thanks again for your help, but unfortunately I am getting more confused with too much irrelevant comments. How about telling us what the actual requirements really actually are? How about explaining why you are using a list? I advised you to use a gb_tree (or something of that sort). Someone else suggested a new-fangled 'map'. Both are good suggestions. What's wrong with them? How many companies are you carrying information about? From on-your-mark@REDACTED Wed Nov 25 06:10:36 2015 From: on-your-mark@REDACTED (YuanZhiqian) Date: Wed, 25 Nov 2015 13:10:36 +0800 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: , <437419A0-880C-4546-98FF-9DA5EFD639D0@cs.otago.ac.nz> , Message-ID: Hi Richard, I am really not interested in quarreling, I hope you are 100%ly clear with this point -- that I HATE to talk with a guy like you who are just showing off his smartness, saying ironic words and showing arrogant attitude. Your aggressive behavior is not welcome here. "tard" is a slang, use a dictionary. So what's your point? Picking out every "illness" in my expression, does it make you feel stronger? Sorry I don't have more patience to look through your verbose comments, learn some respect and be polite before your talk. Read this to solve your "size_t" argument. http://stackoverflow.com/questions/1951519/when-to-use-stdsize-t But I don't think it matter at all, use whatever type you want. And please, leave those arrogant attitude and your cavil to yourself, they are irrelevant to the question. > Well, if I'm not _shown_ the track it's hard to stay on it.I'm afraid this is your own problem. Please keep the discussion on the track and friendly, or just keep silence. Best regardsZhiqian > Subject: Re: [erlang-questions] Update with complicated data structure > From: ok@REDACTED > Date: Wed, 25 Nov 2015 17:13:23 +1300 > CC: erlang-questions@REDACTED > To: on-your-mark@REDACTED > > > On 24/11/2015, at 7:51 pm, YuanZhiqian wrote: > > > Hi Richard, > > > > Thanks for your elaborative comments. Some of the advices are helpful and I appreciate a lot for that; but some of them are not quite on the track ... > > Well, if I'm not _shown_ the track it's hard to stay on it. > > > > For example, regarding the array size issue, you see, it's just a sample where my point is focused on the assignment expression, so I use an array instead of a hash table which I actually have used in my project, for that would make the code messed up with less important parts. I am certainly not a C language tard = = > > "Tard"? What does that mean? > https://en.wikipedia.org/wiki/Tardigrade ? > > > > > And please allow me to clarify my purpose of this question, this is not an erlang v.s. c&c++ debate, > > That was understood; no clarification needed. > > > I am just facing an actual problem in real scenario, and need some practical advice. > > While it's not a C vs Erlang debate, it *is* a matter > of trying to do things the C way in an Erlang program. > > > > So it is not that I insist in making erlang behave the same as imperial language, but that it is a real requirement in the project that I'm trying to solve. > > I doubt that very much. Requirements talk about what a program > does, as seen from outside. How the collection of companies is > held, whether as a list, as a gb tree, as a map, as an ETS > table, or even in the process dictionary (for safety, that > had better be the process dictionary of a separate process) > should be entirely up to you. I have serious trouble > believing in requirements that say > "You must use a data structure that makes search > and update take a long time." > > Your requirements may well say > "Update the consumption of company X by D" > but of course you haven't shown us the requirements. > > One thing may have struck you already. > If you are using a relational data base, > changing one field of one record at a time is > a good way to run quite slowly. One way to get > good performance out of RDBs is *batching*. > > Now if you kept the list of N companies *sorted* on increasing > company_id, and if you had a *batch* of M companies to be > updated, also in a list sorted on increasing company_id, > then you can *merge* those two lists and produce a new list > in O(N+M) time. (This is what old-timers used to do with > tape drives before even CODASYL data bases.) But if you do > it one company at a time, it's going to take O(N*M) time. > If you use a gb_tree, O(lg(N)*M). > > So there is a really serious question about whether you really > *HAVE* to update one field of one record, or whether you have > a *batch* of fields, and possibly multiple updates. > > > > As for the naming issue ... ok you are right, but does it really matter? > > Yes, it does. > > > At least not as important in my case ... > > > > And you are asking me why I am using size_t instead of int, I don't know if I am right, but size_t seems an intended type for array indices, > > (a) It cannot possibly be. C existed long before size_t did. > (b) The name *tells* you what it's meant for: it is meant for > the *sizes* of objects. > (c) Kernighan & Ritchie, "The C programming language", 2nd edition > consistently uses int for array indexing. But what do they > know? They only invented the language. > > > since it is unassigned, > > I think you mean 'unsigned', and that's actually bad. > > > and codes from google use size_t as well. > > Google invented Go. They did not invent C and are > not authorities on how it was meant to be used. > > > Try to compile using int, but configure your error level to the most strict mode, turn on "warning as error" option, then you won't get the code compiled. > > I have no idea what you are talking about. > Code that indexes arrays using int gets through gcc, clang, > lint, and splint at the highest levels with no warnings. > > Good C compilers warn about using *char* as an array index, > because the programmer has no way of telling beforehand > whether char is signed or unsigned, so > a[(char)255] might or might not involve a negative subscript. > But invalidating int would mean breaking almost all the C > code ever written for Unix. > > I just checked the source code for (original) vi, and yep, > it uses int subscripts, including "tp[-1]", which is not > actually an unusual thing to do in C (tp being a pointer > after the beginning of an array), and no usable C compiler is > going to treat that as an error. > > > So the point is, thanks again for your help, but unfortunately I am getting more confused with too much irrelevant comments. > > How about telling us what the actual requirements really > actually are? > How about explaining why you are using a list? > I advised you to use a gb_tree (or something of that sort). > Someone else suggested a new-fangled 'map'. > Both are good suggestions. What's wrong with them? > How many companies are you carrying information about? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Nov 25 06:14:11 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 25 Nov 2015 18:14:11 +1300 Subject: [erlang-questions] Update with complicated data structure In-Reply-To: References: , <437419A0-880C-4546-98FF-9DA5EFD639D0@cs.otago.ac.nz> , Message-ID: <8D46D5DA-53B0-4E35-A06B-C3DCADC8C028@cs.otago.ac.nz> On 25/11/2015, at 6:10 pm, YuanZhiqian wrote: > Hi Richard, > > I am really not interested in quarreling, I hope you are 100%ly clear with this point -- that I HATE to talk with a guy like you who are just showing off his smartness, saying ironic words and showing arrogant attitude. Your aggressive behavior is not welcome here. I am not showing off smartness, using ironic words, or being arrogant or aggressive. > > "tard" is a slang, use a dictionary. I do not know that word. The dictionaries that I did check do not define it either. > > Well, if I'm not _shown_ the track it's hard to stay on it. > I'm afraid this is your own problem. Please keep the discussion on the track and friendly, or just keep silence. But this is the heart of it! YOU STILL HAVE NOT TOLD US WHAT YOUR PROBLEM REALLY IS. You have only told us how you would solve it in C, which is not the same thing at all. And it really matters, because the whole way to structure your code might need to be different. Instead of flaming me, why not address the many positive suggestions that were made? From mjw@REDACTED Wed Nov 25 09:23:49 2015 From: mjw@REDACTED (Michael Wright) Date: Wed, 25 Nov 2015 08:23:49 +0000 Subject: [erlang-questions] IP search optimization In-Reply-To: References: <20151119170510.GA72389@staff.retn.net> Message-ID: I think routers generally use radix trees, and it's certainly important for them to be able to do lookups quickly. Some also have specialised hardware to assist, but I guess you can't have everything! Regarding your desire to use an ETS table for the inheritance, I obviously don't know your exact use case, but I personally feel it's more important to use the correct/optimal storage solution. My reasoning is: 1. Obviously it will perform best. 2. If then you crash, well, fail early is the Erlang way, and hopefully this would lead to a more rapid development of a relatively stable solution. 3. If there is a flaw in your persistent data that caused the crash, you won't persist the flaw and crash again. 4. If it's worth it, you could back up your prefixes in a separate process (or ETS table) anyway. If you backed up your prefixes (point 4) you can reconstruct your radix tree from the backup in the event of a failure. Not perhaps as quick as not having to... but probably not really very expensive unless you really crash very often. Typically routers would do this anyway because they learn routes from various sources and maintain databases for each of those sources and/or each protocol, and then the actual routing table used to make routing decisions is essentially separate, and can be optimised separately. Your use case might be quite different but you could still adopt the idea artificially. Another thing to consider is that you could implement a 'route cache' on top of whatever solution you choose (every time you have to search your radix/ETS stuff the result in a cache, the cache being a map of IP to prefix). This will make most of your lookup speeds very quick and largely constant because they'll be a hash lookup, ASSUMING your use case is such that you get a good cache hit rate. There is additional work to be done of course, you have to garbage collect your cache, you have to invalidate all or some of it when you update your prefix list (unless the lag is acceptable in which case just wait for garbage collection), and you must give very careful consideration to the possibility of dos attacks causing you to build huge caches. This obviously depends on the details of your use case, as does the likelihood of a good cache hit rate. M. On Fri, Nov 20, 2015 at 5:01 PM, Sid Muller wrote: > > Sent: Thursday, November 19, 2015 at 9:05 AM > > From: "Alexandre Snarskii" > > To: "Sid Muller" > > Cc: erlang-questions > > Subject: Re: [erlang-questions] IP search optimization > > > And then I want to check if an IP (31.24.171.44) belongs to any > > > network address in the ets table (the table can get pretty large). > > > > Looks more like a task for radix trees... > > Hmm, good point although with ets if my owner process gets killed I > inherit the ets and no data is lost. Not sure how that cold be done with a > radix tree. > > > > The best you can achieve with ets is: > > on insert: make sure that inserted networks are normalized (do not have > > any bits set after prefix length). > > on lookup: > > 1. start with masklen of 32 > > 2. normalize address to current masklen > > 3. lookup for address/masklen record, return if found > > 4. if current masklen is zero - return 'not_found' > > 5. decrease masklen and repeat from step two. > > There are a few faster methods: > 1. Have a pre-scanned list of /CIDR [21, 20...], there are usually about > 10. But this still traverses the table 10 times in worst case. > > 2. Traverse the table with ets:lookup(ets:next(), this has the advantage > over #1 as it's much faster in most cases. > > 3. ets:select(MatchSpec), this is probably the fastest solution because it > should traverse the table only once. But right now I'm having a bit of a > problem with my select statement... see other email [bitstring select] > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Wed Nov 25 09:41:12 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Wed, 25 Nov 2015 09:41:12 +0100 Subject: [erlang-questions] Would like some feedback In-Reply-To: References: <5654AA10.3040007@xs4all.nl> Message-ID: On making prevkeys.erl (or should I say axolotl_prevkeys.erl) more readable: Match your records in your function head instead of extracting their fields one by one. Create a session_data record that could be exported as an opaque type and implement accessors. Typically this would go in its own module. Nice work Cheers, -- Pierre Fenoll On 25 November 2015 at 00:55, Chandru wrote: > Hi Frans, > > Here are a few comments with the caveat that I haven't spent any time > understanding the protocol. So the comments are more about implementation > style... > > - Firstly, its quite impressive you've managed to write this in quite an > idiomatic style for your hobby project, so well done! > - You might want to consider naming all your modules with the 'axolotl_' > prefix to minimise the possibility of module name clashes when others use > your code > - It would be a good idea to put all your DB access operations in a module > such as axolotl_db.erl so that if you ever wanted to changed your database > backend, the changes are limited to one module. It might not be that > important for this particular application, but as rule of thumb it will > serve you well. > - You raise a good question about "... decide what goes into a new process > or is done in the calling process...". I'll try and clarify this with an > example. You have implemented identities:check_registration/2 as a > gen_server:call, but the code which handles this message in the handle_call > function is just reading from mnesia and returning a result. This need not > be a gen_server:call, it can be written as follows. > > check(Registration_id, Dhi_pub) -> > case mnesia:dirty_read(identity, Dhi_pub) of > [#identity{registration_id = Registration_id}] -> > true; > _ -> > false > end. > > This is because there is no value in putting it through the gen_server. It > still makes sense for the code to reside in this module as it logically > belongs here. > > - Typically with Erlang applications, your mnesia initialisation code > should be separate from your normal running code. It is okay to expect the > user to run a one off installation function to create the required tables. > Also, it is not a good idea to delete schema and recreate it because it > becomes hard to use this as part of a node with other applications which > have their own mnesia tables. > > - The code in prekeys.erl is a bit hard to read. You might want to move > the code from each of the handle_call function clauses into separate > functions which are invoked. This is probably partly due to my ignorance of > the protocol, but the code could be made to look cleaner. > > - I couldn't see any test code in the modules, how are you testing this? > > I hope this is of some help. > > cheers, > Chandru > > > > On 24 November 2015 at 18:18, Frans Schneider wrote: > >> Dear list, >> >> I've been learning Erlang in my spare-time for some time now and would >> love some feedback on a project I recently started. Since I have no >> coworkers or other people to get feedback from, maybe somebody on this list >> could have a quick look. >> The things I find particular difficult is defining a proper API, to >> decide what goes into a new process or is done in the calling process and >> dealing with errors. I have no formal training as a programmer which may be >> the reason I have trouble to get these things right. >> >> The project I took [1] was to implement the Axolotl ratcheting protocol >> from Open Whisper Systems [2]. The protocol [3] itself didn't look very >> difficult, but since there are no proper specs for the total system, I used >> the libaxolotl-java implementation as a starting pont [4]. Reading the Java >> code probably was the hardest part of the whole project. >> >> I think I covered most of the functionality of the libaxolotl-java >> library. I haven't verified interoperability between my implementation and >> libaxolotl-java since that would require me to start coding in Java which >> is not something I am looking forward to. >> Comparing my Erlang code with the original Java code, I needed far less >> lines of code and, I think, the code is much easier to understand. Also, >> building and deploying the code seems much simpler. >> >> So, any comments are most welcome. >> >> Thanks, >> >> Frans >> >> [1] https://github.com/schnef/axolotl >> [2] https://www.whispersystems.org/ >> [3] https://github.com/trevp/axolotl/wiki >> [4] https://github.com/WhisperSystems/libaxolotl-java >> Nice implementation https://github.com/rxcomm/pyaxo >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Wed Nov 25 11:57:31 2015 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Wed, 25 Nov 2015 11:57:31 +0100 Subject: [erlang-questions] [ANN] geas 2.0.0-alpha Message-ID: <5655941B.2080003@wanadoo.fr> Hi list, I'm hereby announcing an alpha-release of the geas 2.0 module. https://github.com/crownedgrouse/geas A major version change for a major new feature. Geas can now give the compatibility of beam files with official Erlang releases from R15B to 18.1 . For example, using application:ensure_all_started/1 can only be used starting R16B01 or maps starting 17.0. As well pg module cannot be used after 17.5. Geas extract all the function calls from abstract code (if available) and compare to its database [1]. Geas database is a .hrl file containing all modules and exported functions informations from official releases (even undocumented ones) with their starting and eventually ending version. This database is created from incremental Erlang release difference inventory [2] Geas gives you finally the official Erlang release window where the beam files are runnable. So it can help you to : - Write your Travis CI (or equivalent) config file - Know if beam files can run with an Erlang VM - Verify that your dependencies can run on same Erlang release window than your project - Write clear statements in your README project files Note, it is not a guarantee that beam files will run correctly on the whole release window (some bugs may exists in older release for instance). But this will tell you it cannot work correctly outside this window. As well Geas can inform you on what are the functions which reduce the Erlang release window, so it can then help you to : - Modify your code in order to increase runnable release window - Know if application/module have to be fixed in order to run on the last official release In this alpha version, only a raw analyze is done of beam files and no attempt is done to understand the code. So, even if a function is protected with a 'catch', it will be part of the analyze. Further work will be needed to add a smarter analyze of code. However I plan to add some ways to ignore some module/function if necessary. I hope you will enjoy Geas 2.0 ! Feedback is welcome... Regards PS : Why starting from R15 ? Because older releases cannot compile on my machines. But if some people are still using older releases and wanting to contribute to the project, it is very easy. Contact me in such case. [1] https://github.com/crownedgrouse/geas/blob/master/src/geas_db.hrl [2] https://github.com/crownedgrouse/geas/tree/master/doc/reldiffs (available in Erlang term and Yaml format) From jesper.louis.andersen@REDACTED Wed Nov 25 13:13:54 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 25 Nov 2015 13:13:54 +0100 Subject: [erlang-questions] Would like some feedback In-Reply-To: <5654AA10.3040007@xs4all.nl> References: <5654AA10.3040007@xs4all.nl> Message-ID: On Tue, Nov 24, 2015 at 7:18 PM, Frans Schneider wrote: > So, any comments are most welcome. Small nitpick as well: You want to put your curve25519 and hkdf libraries as dependencies in the .app.src. If you build a release with axolotl, then this is needed to automatically include those dependencies, and they are necessary to make the program run. You also need to supply any other dependencies, like mnesia there. As Sergej writes, you can win a lot by splitting off dependencies on mnesia and also split protocol encoding/decoding from session handling. That way, it becomes easier to adapt the protocol to other systems. When you have easy concurrency, it is often alluring to use it in every project you see. But beware, since it also tend to bind you into a specific architecture. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Nov 26 05:52:18 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 26 Nov 2015 17:52:18 +1300 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice Message-ID: Summary: (1) A promising way to solve the nested updates problem is to be more functional, not less. (2) There's an approach we could steal from Haskell. (3) There's free code to play with, worth every penny. Consider a simplified version of Pascal: ::= * ::= | | ::= '^' ::= '.' ::= '[' ']' If we think of a selector as a function f :: var T1 -> var T2 it's clear that is function application and is function composition. Yet in Pascal, and C, and languages like them, while you can *apply* selectors and *compose* them, they cannot be passed as parameters or even named. If a functional language has something like selectors, they should be first class values, just like other more-or-less-function-like things. But what kind of value? I think it was Reynolds, analysing Algol 60, who came up with the idea that a pass-by-name parameter was really a *pair* of functions: one to fetch a value and one to store it, and thereby freed us from the limiting idea that there needed to be a 'var T' type. The Haskell community have taken this idea and developed it. There this concept is called a "Lens". http://www.cs.otago.ac.nz/staffpriv/ok/lens.erl is a crude implementation of lenses in Erlang. A lens is a selector from (values of) type A to (values of) type B. It's represented as a triple {Get, Put, Upd} Get :: (A) -> B Put :: (A, B) -> A Upd :: (A, (B) -> B) -> A The Get function says how to extract a B value from an A value. The Put function says how to put a B value into an existing A value, returning a new A value. The Upd function says how to compute a new A value by extracting a B, applying a function to it, and putting it back. We expect the following laws to hold: Get(Put(A, B)) = B Put(A, Get(A)) = A Upd(A, Fun) = Put(A, Fun(Get(A))) That is, if you make a lens by hand, you should ensure that this is true. There are functions for using lenses so you don't have to think about the representation. lens:get(Lens, A) -> B lens:put(Lens, A, B) -> A' lens:update(Lens, A, F) -> A' There are also functions for making simple lenses. For example, if you have a record 'rec' with a field 'fld', lens:tuple(#rec.fld) will give you a lens for that field. Warning: the functions in the lens returned by lens:tuple(#rec.fld) do NOT check at run time that they are dealing with a 'rec' record. They can't, because #rec.fld is just a number, and you can't get back from that number to 'rec' or the arity of the record. The important thing is that these functions compose. For example, suppose you have -record(foo, {boo,coo,goo,zoo}). and Foozle is a list of these, and you want to find the first record with boo=42, and increment its zoo field by 137. Find = lens:where(fun (#foo{boo=Boo}) -> Boo == 42 end) is a lens for finding the record you want and Field = lens:tuple(#foo.zoo) is a lens for getting at the zoo field of such a record. lens.erl provides composition of 2 .. 6 lenses. So Selector = lens:c(Find, Field) is a selector that refers to a field of a record in a list, and lens:update(Selector, Foozle, fun (Zoo) -> Zoo + 137 end) does the whole job (in just one pass through the list, too, which is why the Upd function is there). More simply, lens:c(lens:tuple(#a.b), lens:tuple(#c.d), lens:tuple(#e.f), lens:tuple(#g.h)) is the equivalent of .b.d.f.h in Pascal, except that it is a value you can pass around like any other. You are not limited to fields that actually exist as such. For example, you could do gb_set(Element) -> { fun (Set) -> gb_sets:is_member(Element, Set) end , fun (Set, false) -> gb_sets:delete_any(Element, Set) ; (Set, true) -> gb_sets:add(Element, Set) end , fun (Set, Fun) -> Old = gb_sets:is_member(Element, Set), case Fun(Old) of Old -> Set ; true -> gb_sets:insert(Element, Set) ; false -> gb_sets:delete(Element, Set) end end }. and make a gb-set act like a dictionary with Boolean elements. There's even a combinator all/1 so that All = lens:c(lens:tuple(1)), Data = [{3,a},{1,b},{4,c},{6,d}] lens:get(All, Data) => [3,1,4,6] lens:update(All, Data, fun (N) -> N*10 end) => [{30,a},{10,b},{40,c},{60,d}] So what about cross-module inlining? By the time you've made and composed a couple of lenses, you have a tangle of funs. If the compiler (a) always inlines (fun (...) -> ... end)(...) and (b) inlines the functions from the lens module, then the tangle resolves tidily into reasonably straightforward code. But the cross-module inlining step is crucial. Without that, the compiler has nothing to work on. At its crudest, consider lens:get(lens:tuple(#rec.fld), Rec) *With* cross-module inlining and inlining of funs that are applied to known arguments and dead code elimination, you end up with element(#rec.fld, Rec) *Without* cross-module inlining, three funs are built, only one of which will ever be called. From bchesneau@REDACTED Thu Nov 26 11:37:20 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 26 Nov 2015 10:37:20 +0000 Subject: [erlang-questions] pattern to augment a release with a plugin system Message-ID: Hi all, I am actually working on a system to augment a release possibly dynamically with some extensions. For now in hooks [1] I am starting applications from the filesystem or just using the applications shipped with the release. I am not a fan of simply starting/stopping applications from the filesystem since for now it is not handling upgrades (only by starting/stopping the application) like a release can do. I can see these different ways to handle plugins right now: 1) plugins are only added at compile time (something rabbitmq does). Similar to 1 pro: plugins are the simple apps cons: - the system is not dynamic and a user can't simply augment the application he is using. Only plugins built and shipped with the release are available 2) adding plugins to a release and possibly starting them when enabled via the config. The upgrade would be done via a release upgrade. Pro: - doesn't change too much the current way to handle releases upgrade. - the plugins can be added dynamically Cons: - if i want to add a new plugin I will need to create a new release, increment its version etc. - Also a new release would have to be build for each users depending on their choice. - I have to find a way to build a local release based on remote plugins depending on the platform 3) load plugins via an external system. If this is an apps, it will be installed using the release_handler, so it could possibly manage the .appup scripts pro: plugins are really dynamically handled cons: - not sure on how with the release_handler the installed application will be handled at startup. - not sure if dynamically augmented release using the release_handler is suppored 4) Something similar to 3 , but instead of using the release_handler, i have my own plugin_handler handling upgrades by looking on specific modules functions if needed. Are there other ways to do it? What would be the right pattern to do it? To be honest I am balancing between 2 and 3. 4 would be fine but would mean rewriting something similar to the release_handler. Generally speaking how people feels about augmenting dynamically a release with new applications/modules ? Any feedback is welcome. - beno?t [1] https://github.com/barrel-db/hooks -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Thu Nov 26 11:42:02 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Thu, 26 Nov 2015 11:42:02 +0100 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: References: Message-ID: That's something the JIT should be doing, right? Anyhow, what happens when inlined code throws an error? Which location information is included? I'm thinking "if none" then "inline all the funs" away. Cheers, -- Pierre Fenoll On 26 November 2015 at 05:52, Richard A. O'Keefe wrote: > Summary: > (1) A promising way to solve the nested updates problem > is to be more functional, not less. > (2) There's an approach we could steal from Haskell. > (3) There's free code to play with, worth every penny. > > Consider a simplified version of Pascal: > > ::= * > ::= | | > ::= '^' > ::= '.' > ::= '[' ']' > > If we think of a selector as a function > f :: var T1 -> var T2 > it's clear that is function > application and is > function composition. > > Yet in Pascal, and C, and languages like them, > while you can *apply* selectors and *compose* them, > they cannot be passed as parameters or even named. > > If a functional language has something like selectors, > they should be first class values, just like other > more-or-less-function-like things. But what kind of > value? > > I think it was Reynolds, analysing Algol 60, who came > up with the idea that a pass-by-name parameter was > really a *pair* of functions: one to fetch a value > and one to store it, and thereby freed us from the > limiting idea that there needed to be a 'var T' type. > > The Haskell community have taken this idea and > developed it. There this concept is called a "Lens". > > http://www.cs.otago.ac.nz/staffpriv/ok/lens.erl > > is a crude implementation of lenses in Erlang. > A lens is a selector from (values of) type A to > (values of) type B. It's represented as a triple > > {Get, Put, Upd} > Get :: (A) -> B > Put :: (A, B) -> A > Upd :: (A, (B) -> B) -> A > > The Get function says how to extract a B value from an > A value. The Put function says how to put a B value > into an existing A value, returning a new A value. > The Upd function says how to compute a new A value by > extracting a B, applying a function to it, and putting > it back. We expect the following laws to hold: > > Get(Put(A, B)) = B > Put(A, Get(A)) = A > Upd(A, Fun) = Put(A, Fun(Get(A))) > > That is, if you make a lens by hand, you should ensure > that this is true. > > There are functions for using lenses so you don't have > to think about the representation. > > lens:get(Lens, A) -> B > lens:put(Lens, A, B) -> A' > lens:update(Lens, A, F) -> A' > > There are also functions for making simple lenses. > For example, if you have a record 'rec' with a field > 'fld', lens:tuple(#rec.fld) will give you a lens > for that field. > > Warning: the functions in the lens returned by > lens:tuple(#rec.fld) > do NOT check at run time that they are dealing with > a 'rec' record. They can't, because #rec.fld is just > a number, and you can't get back from that number to > 'rec' or the arity of the record. > > The important thing is that these functions compose. > > For example, suppose you have > -record(foo, {boo,coo,goo,zoo}). > and Foozle is a list of these, and you want to find > the first record with boo=42, and increment its zoo > field by 137. > > Find = lens:where(fun (#foo{boo=Boo}) -> Boo == 42 end) > > is a lens for finding the record you want and > > Field = lens:tuple(#foo.zoo) > > is a lens for getting at the zoo field of such a record. > lens.erl provides composition of 2 .. 6 lenses. So > > Selector = lens:c(Find, Field) > > is a selector that refers to a field of a record in a list, > and > > lens:update(Selector, Foozle, fun (Zoo) -> Zoo + 137 end) > > does the whole job (in just one pass through the list, too, > which is why the Upd function is there). > > More simply, > lens:c(lens:tuple(#a.b), lens:tuple(#c.d), > lens:tuple(#e.f), lens:tuple(#g.h)) > is the equivalent of .b.d.f.h in Pascal, except that it is > a value you can pass around like any other. > > You are not limited to fields that actually exist as such. > For example, you could do > > gb_set(Element) -> > { fun (Set) -> gb_sets:is_member(Element, Set) end > , fun (Set, false) -> gb_sets:delete_any(Element, Set) > ; (Set, true) -> gb_sets:add(Element, Set) > end > , fun (Set, Fun) -> > Old = gb_sets:is_member(Element, Set), > case Fun(Old) > of Old -> Set > ; true -> gb_sets:insert(Element, Set) > ; false -> gb_sets:delete(Element, Set) > end > end > }. > > and make a gb-set act like a dictionary with Boolean elements. > > There's even a combinator all/1 so that > All = lens:c(lens:tuple(1)), > Data = [{3,a},{1,b},{4,c},{6,d}] > lens:get(All, Data) > => [3,1,4,6] > lens:update(All, Data, fun (N) -> N*10 end) > => [{30,a},{10,b},{40,c},{60,d}] > > So what about cross-module inlining? > > By the time you've made and composed a couple of lenses, > you have a tangle of funs. If the compiler (a) always > inlines (fun (...) -> ... end)(...) and (b) inlines the > functions from the lens module, then the tangle resolves > tidily into reasonably straightforward code. > > But the cross-module inlining step is crucial. Without > that, the compiler has nothing to work on. > > At its crudest, consider > > lens:get(lens:tuple(#rec.fld), Rec) > > *With* cross-module inlining and inlining of funs that > are applied to known arguments and dead code elimination, > you end up with > > element(#rec.fld, Rec) > > *Without* cross-module inlining, three funs are built, > only one of which will ever be called. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Thu Nov 26 14:35:49 2015 From: t@REDACTED (Tristan Sloughter) Date: Thu, 26 Nov 2015 07:35:49 -0600 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: References: Message-ID: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Jesper also created https://github.com/jlouis/erl-lenses a few years back for this. -- Tristan Sloughter t@REDACTED On Wed, Nov 25, 2015, at 10:52 PM, Richard A. O'Keefe wrote: > Summary: > (1) A promising way to solve the nested updates problem > is to be more functional, not less. > (2) There's an approach we could steal from Haskell. > (3) There's free code to play with, worth every penny. > > Consider a simplified version of Pascal: > > ::= * > ::= | | > ::= '^' > ::= '.' > ::= '[' ']' > > If we think of a selector as a function > f :: var T1 -> var T2 > it's clear that is function > application and is > function composition. > > Yet in Pascal, and C, and languages like them, > while you can *apply* selectors and *compose* them, > they cannot be passed as parameters or even named. > > If a functional language has something like selectors, > they should be first class values, just like other > more-or-less-function-like things. But what kind of > value? > > I think it was Reynolds, analysing Algol 60, who came > up with the idea that a pass-by-name parameter was > really a *pair* of functions: one to fetch a value > and one to store it, and thereby freed us from the > limiting idea that there needed to be a 'var T' type. > > The Haskell community have taken this idea and > developed it. There this concept is called a "Lens". > > http://www.cs.otago.ac.nz/staffpriv/ok/lens.erl > > is a crude implementation of lenses in Erlang. > A lens is a selector from (values of) type A to > (values of) type B. It's represented as a triple > > {Get, Put, Upd} > Get :: (A) -> B > Put :: (A, B) -> A > Upd :: (A, (B) -> B) -> A > > The Get function says how to extract a B value from an > A value. The Put function says how to put a B value > into an existing A value, returning a new A value. > The Upd function says how to compute a new A value by > extracting a B, applying a function to it, and putting > it back. We expect the following laws to hold: > > Get(Put(A, B)) = B > Put(A, Get(A)) = A > Upd(A, Fun) = Put(A, Fun(Get(A))) > > That is, if you make a lens by hand, you should ensure > that this is true. > > There are functions for using lenses so you don't have > to think about the representation. > > lens:get(Lens, A) -> B > lens:put(Lens, A, B) -> A' > lens:update(Lens, A, F) -> A' > > There are also functions for making simple lenses. > For example, if you have a record 'rec' with a field > 'fld', lens:tuple(#rec.fld) will give you a lens > for that field. > > Warning: the functions in the lens returned by > lens:tuple(#rec.fld) > do NOT check at run time that they are dealing with > a 'rec' record. They can't, because #rec.fld is just > a number, and you can't get back from that number to > 'rec' or the arity of the record. > > The important thing is that these functions compose. > > For example, suppose you have > -record(foo, {boo,coo,goo,zoo}). > and Foozle is a list of these, and you want to find > the first record with boo=42, and increment its zoo > field by 137. > > Find = lens:where(fun (#foo{boo=Boo}) -> Boo == 42 end) > > is a lens for finding the record you want and > > Field = lens:tuple(#foo.zoo) > > is a lens for getting at the zoo field of such a record. > lens.erl provides composition of 2 .. 6 lenses. So > > Selector = lens:c(Find, Field) > > is a selector that refers to a field of a record in a list, > and > > lens:update(Selector, Foozle, fun (Zoo) -> Zoo + 137 end) > > does the whole job (in just one pass through the list, too, > which is why the Upd function is there). > > More simply, > lens:c(lens:tuple(#a.b), lens:tuple(#c.d), > lens:tuple(#e.f), lens:tuple(#g.h)) > is the equivalent of .b.d.f.h in Pascal, except that it is > a value you can pass around like any other. > > You are not limited to fields that actually exist as such. > For example, you could do > > gb_set(Element) -> > { fun (Set) -> gb_sets:is_member(Element, Set) end > , fun (Set, false) -> gb_sets:delete_any(Element, Set) > ; (Set, true) -> gb_sets:add(Element, Set) > end > , fun (Set, Fun) -> > Old = gb_sets:is_member(Element, Set), > case Fun(Old) > of Old -> Set > ; true -> gb_sets:insert(Element, Set) > ; false -> gb_sets:delete(Element, Set) > end > end > }. > > and make a gb-set act like a dictionary with Boolean elements. > > There's even a combinator all/1 so that > All = lens:c(lens:tuple(1)), > Data = [{3,a},{1,b},{4,c},{6,d}] > lens:get(All, Data) > => [3,1,4,6] > lens:update(All, Data, fun (N) -> N*10 end) > => [{30,a},{10,b},{40,c},{60,d}] > > So what about cross-module inlining? > > By the time you've made and composed a couple of lenses, > you have a tangle of funs. If the compiler (a) always > inlines (fun (...) -> ... end)(...) and (b) inlines the > functions from the lens module, then the tangle resolves > tidily into reasonably straightforward code. > > But the cross-module inlining step is crucial. Without > that, the compiler has nothing to work on. > > At its crudest, consider > > lens:get(lens:tuple(#rec.fld), Rec) > > *With* cross-module inlining and inlining of funs that > are applied to known arguments and dead code elimination, > you end up with > > element(#rec.fld, Rec) > > *Without* cross-module inlining, three funs are built, > only one of which will ever be called. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Thu Nov 26 15:14:20 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 26 Nov 2015 15:14:20 +0100 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Message-ID: It has the same problems as Richard alludes to: in order to run fast, you need functional optimizations to remove the intermediate functions and make access fast. I toyed with building a parse transform for producing stuff, but I'd much rather want a JIT. On Thu, Nov 26, 2015 at 2:35 PM, Tristan Sloughter wrote: > Jesper also created https://github.com/jlouis/erl-lenses a few years > back for this. > > -- > Tristan Sloughter > t@REDACTED > > On Wed, Nov 25, 2015, at 10:52 PM, Richard A. O'Keefe wrote: > > Summary: > > (1) A promising way to solve the nested updates problem > > is to be more functional, not less. > > (2) There's an approach we could steal from Haskell. > > (3) There's free code to play with, worth every penny. > > > > Consider a simplified version of Pascal: > > > > ::= * > > ::= | | > > ::= '^' > > ::= '.' > > ::= '[' ']' > > > > If we think of a selector as a function > > f :: var T1 -> var T2 > > it's clear that is function > > application and is > > function composition. > > > > Yet in Pascal, and C, and languages like them, > > while you can *apply* selectors and *compose* them, > > they cannot be passed as parameters or even named. > > > > If a functional language has something like selectors, > > they should be first class values, just like other > > more-or-less-function-like things. But what kind of > > value? > > > > I think it was Reynolds, analysing Algol 60, who came > > up with the idea that a pass-by-name parameter was > > really a *pair* of functions: one to fetch a value > > and one to store it, and thereby freed us from the > > limiting idea that there needed to be a 'var T' type. > > > > The Haskell community have taken this idea and > > developed it. There this concept is called a "Lens". > > > > http://www.cs.otago.ac.nz/staffpriv/ok/lens.erl > > > > is a crude implementation of lenses in Erlang. > > A lens is a selector from (values of) type A to > > (values of) type B. It's represented as a triple > > > > {Get, Put, Upd} > > Get :: (A) -> B > > Put :: (A, B) -> A > > Upd :: (A, (B) -> B) -> A > > > > The Get function says how to extract a B value from an > > A value. The Put function says how to put a B value > > into an existing A value, returning a new A value. > > The Upd function says how to compute a new A value by > > extracting a B, applying a function to it, and putting > > it back. We expect the following laws to hold: > > > > Get(Put(A, B)) = B > > Put(A, Get(A)) = A > > Upd(A, Fun) = Put(A, Fun(Get(A))) > > > > That is, if you make a lens by hand, you should ensure > > that this is true. > > > > There are functions for using lenses so you don't have > > to think about the representation. > > > > lens:get(Lens, A) -> B > > lens:put(Lens, A, B) -> A' > > lens:update(Lens, A, F) -> A' > > > > There are also functions for making simple lenses. > > For example, if you have a record 'rec' with a field > > 'fld', lens:tuple(#rec.fld) will give you a lens > > for that field. > > > > Warning: the functions in the lens returned by > > lens:tuple(#rec.fld) > > do NOT check at run time that they are dealing with > > a 'rec' record. They can't, because #rec.fld is just > > a number, and you can't get back from that number to > > 'rec' or the arity of the record. > > > > The important thing is that these functions compose. > > > > For example, suppose you have > > -record(foo, {boo,coo,goo,zoo}). > > and Foozle is a list of these, and you want to find > > the first record with boo=42, and increment its zoo > > field by 137. > > > > Find = lens:where(fun (#foo{boo=Boo}) -> Boo == 42 end) > > > > is a lens for finding the record you want and > > > > Field = lens:tuple(#foo.zoo) > > > > is a lens for getting at the zoo field of such a record. > > lens.erl provides composition of 2 .. 6 lenses. So > > > > Selector = lens:c(Find, Field) > > > > is a selector that refers to a field of a record in a list, > > and > > > > lens:update(Selector, Foozle, fun (Zoo) -> Zoo + 137 end) > > > > does the whole job (in just one pass through the list, too, > > which is why the Upd function is there). > > > > More simply, > > lens:c(lens:tuple(#a.b), lens:tuple(#c.d), > > lens:tuple(#e.f), lens:tuple(#g.h)) > > is the equivalent of .b.d.f.h in Pascal, except that it is > > a value you can pass around like any other. > > > > You are not limited to fields that actually exist as such. > > For example, you could do > > > > gb_set(Element) -> > > { fun (Set) -> gb_sets:is_member(Element, Set) end > > , fun (Set, false) -> gb_sets:delete_any(Element, Set) > > ; (Set, true) -> gb_sets:add(Element, Set) > > end > > , fun (Set, Fun) -> > > Old = gb_sets:is_member(Element, Set), > > case Fun(Old) > > of Old -> Set > > ; true -> gb_sets:insert(Element, Set) > > ; false -> gb_sets:delete(Element, Set) > > end > > end > > }. > > > > and make a gb-set act like a dictionary with Boolean elements. > > > > There's even a combinator all/1 so that > > All = lens:c(lens:tuple(1)), > > Data = [{3,a},{1,b},{4,c},{6,d}] > > lens:get(All, Data) > > => [3,1,4,6] > > lens:update(All, Data, fun (N) -> N*10 end) > > => [{30,a},{10,b},{40,c},{60,d}] > > > > So what about cross-module inlining? > > > > By the time you've made and composed a couple of lenses, > > you have a tangle of funs. If the compiler (a) always > > inlines (fun (...) -> ... end)(...) and (b) inlines the > > functions from the lens module, then the tangle resolves > > tidily into reasonably straightforward code. > > > > But the cross-module inlining step is crucial. Without > > that, the compiler has nothing to work on. > > > > At its crudest, consider > > > > lens:get(lens:tuple(#rec.fld), Rec) > > > > *With* cross-module inlining and inlining of funs that > > are applied to known arguments and dead code elimination, > > you end up with > > > > element(#rec.fld, Rec) > > > > *Without* cross-module inlining, three funs are built, > > only one of which will ever be called. > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Thu Nov 26 15:20:47 2015 From: schneider@REDACTED (Frans Schneider) Date: Thu, 26 Nov 2015 15:20:47 +0100 Subject: [erlang-questions] Would like some feedback In-Reply-To: References: <5654AA10.3040007@xs4all.nl> Message-ID: <5657153F.4060609@xs4all.nl> Thank you all for taking the time to look at my code. Very much appreciated! Cheers, Frans On 11/25/2015 01:13 PM, Jesper Louis Andersen wrote: > > On Tue, Nov 24, 2015 at 7:18 PM, Frans Schneider > wrote: > > So, any comments are most welcome. > > > Small nitpick as well: You want to put your curve25519 and hkdf > libraries as dependencies in the .app.src. If you build a release with > axolotl, then this is needed to automatically include those > dependencies, and they are necessary to make the program run. > > You also need to supply any other dependencies, like mnesia there. > > As Sergej writes, you can win a lot by splitting off dependencies on > mnesia and also split protocol encoding/decoding from session > handling. That way, it becomes easier to adapt the protocol to other > systems. When you have easy concurrency, it is often alluring to use > it in every project you see. But beware, since it also tend to bind > you into a specific architecture. > > > > -- > J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangud@REDACTED Thu Nov 26 17:00:29 2015 From: dangud@REDACTED (Dan Gudmundsson) Date: Thu, 26 Nov 2015 16:00:29 +0000 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Message-ID: Nice Maybe a stupid question I have not looked at lenses before but why include the "path of keys" into the lens? Why not make them reusable and define get({G,_,_}, Key, X) -> G(Key, X). put({_,P,_}, Key, X, Y) -> P(Key, X, Y). update({_,_,U}, Key, X, F) -> U(Key, X, F). tuple() -> { fun (N, T) -> element(N, T) end , fun (N, T, Y) -> setelement(N, T, Y) end , fun (N, T, F) -> setelement(N, T, F(element(N, T))) end }. list() -> { fun list_get/2, fun list_put/3, fun list_update/3 }. list_get(hd, [H|_]) -> H; list_get(tl, [_|T]) -> T; list_get(N, List) -> nth(N, List). <..cut..> c({G1,P1,U1}, {G2,P2,U2}) -> { fun ([K1,K2], F0) -> G2(K2,G1(K1,F0)) end , fun ([K1,K2], F0, R2) -> F1 = G1(K1, F0), R1 = P2(K2, F1, R2), P1(K1, F0, R1) end , fun ([K1,K2], F0, UF) -> U1(K1, F0, fun (F1) -> U2(K2, F1, UF) end) end }. ... > lens:update(lens:c(lens:tuple(), lens:list()), [2,hd], {[1,2], [a,b]}, fun(Old) -> {Old, new} end). {[1,2],[{a,new},b]} Another question, should/could we add a fourth fun (map) for traversal? As it is all/1 and where/1 assumes lists and does not operate on for example gb_trees or dict. On Thu, Nov 26, 2015 at 3:15 PM Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > It has the same problems as Richard alludes to: in order to run fast, you > need functional optimizations to remove the intermediate functions and make > access fast. > > I toyed with building a parse transform for producing stuff, but I'd much > rather want a JIT. > > > On Thu, Nov 26, 2015 at 2:35 PM, Tristan Sloughter > wrote: > >> Jesper also created https://github.com/jlouis/erl-lenses a few years >> back for this. >> >> -- >> Tristan Sloughter >> t@REDACTED >> >> On Wed, Nov 25, 2015, at 10:52 PM, Richard A. O'Keefe wrote: >> > Summary: >> > (1) A promising way to solve the nested updates problem >> > is to be more functional, not less. >> > (2) There's an approach we could steal from Haskell. >> > (3) There's free code to play with, worth every penny. >> > >> > Consider a simplified version of Pascal: >> > >> > ::= * >> > ::= | | >> > ::= '^' >> > ::= '.' >> > ::= '[' ']' >> > >> > If we think of a selector as a function >> > f :: var T1 -> var T2 >> > it's clear that is function >> > application and is >> > function composition. >> > >> > Yet in Pascal, and C, and languages like them, >> > while you can *apply* selectors and *compose* them, >> > they cannot be passed as parameters or even named. >> > >> > If a functional language has something like selectors, >> > they should be first class values, just like other >> > more-or-less-function-like things. But what kind of >> > value? >> > >> > I think it was Reynolds, analysing Algol 60, who came >> > up with the idea that a pass-by-name parameter was >> > really a *pair* of functions: one to fetch a value >> > and one to store it, and thereby freed us from the >> > limiting idea that there needed to be a 'var T' type. >> > >> > The Haskell community have taken this idea and >> > developed it. There this concept is called a "Lens". >> > >> > http://www.cs.otago.ac.nz/staffpriv/ok/lens.erl >> > >> > is a crude implementation of lenses in Erlang. >> > A lens is a selector from (values of) type A to >> > (values of) type B. It's represented as a triple >> > >> > {Get, Put, Upd} >> > Get :: (A) -> B >> > Put :: (A, B) -> A >> > Upd :: (A, (B) -> B) -> A >> > >> > The Get function says how to extract a B value from an >> > A value. The Put function says how to put a B value >> > into an existing A value, returning a new A value. >> > The Upd function says how to compute a new A value by >> > extracting a B, applying a function to it, and putting >> > it back. We expect the following laws to hold: >> > >> > Get(Put(A, B)) = B >> > Put(A, Get(A)) = A >> > Upd(A, Fun) = Put(A, Fun(Get(A))) >> > >> > That is, if you make a lens by hand, you should ensure >> > that this is true. >> > >> > There are functions for using lenses so you don't have >> > to think about the representation. >> > >> > lens:get(Lens, A) -> B >> > lens:put(Lens, A, B) -> A' >> > lens:update(Lens, A, F) -> A' >> > >> > There are also functions for making simple lenses. >> > For example, if you have a record 'rec' with a field >> > 'fld', lens:tuple(#rec.fld) will give you a lens >> > for that field. >> > >> > Warning: the functions in the lens returned by >> > lens:tuple(#rec.fld) >> > do NOT check at run time that they are dealing with >> > a 'rec' record. They can't, because #rec.fld is just >> > a number, and you can't get back from that number to >> > 'rec' or the arity of the record. >> > >> > The important thing is that these functions compose. >> > >> > For example, suppose you have >> > -record(foo, {boo,coo,goo,zoo}). >> > and Foozle is a list of these, and you want to find >> > the first record with boo=42, and increment its zoo >> > field by 137. >> > >> > Find = lens:where(fun (#foo{boo=Boo}) -> Boo == 42 end) >> > >> > is a lens for finding the record you want and >> > >> > Field = lens:tuple(#foo.zoo) >> > >> > is a lens for getting at the zoo field of such a record. >> > lens.erl provides composition of 2 .. 6 lenses. So >> > >> > Selector = lens:c(Find, Field) >> > >> > is a selector that refers to a field of a record in a list, >> > and >> > >> > lens:update(Selector, Foozle, fun (Zoo) -> Zoo + 137 end) >> > >> > does the whole job (in just one pass through the list, too, >> > which is why the Upd function is there). >> > >> > More simply, >> > lens:c(lens:tuple(#a.b), lens:tuple(#c.d), >> > lens:tuple(#e.f), lens:tuple(#g.h)) >> > is the equivalent of .b.d.f.h in Pascal, except that it is >> > a value you can pass around like any other. >> > >> > You are not limited to fields that actually exist as such. >> > For example, you could do >> > >> > gb_set(Element) -> >> > { fun (Set) -> gb_sets:is_member(Element, Set) end >> > , fun (Set, false) -> gb_sets:delete_any(Element, Set) >> > ; (Set, true) -> gb_sets:add(Element, Set) >> > end >> > , fun (Set, Fun) -> >> > Old = gb_sets:is_member(Element, Set), >> > case Fun(Old) >> > of Old -> Set >> > ; true -> gb_sets:insert(Element, Set) >> > ; false -> gb_sets:delete(Element, Set) >> > end >> > end >> > }. >> > >> > and make a gb-set act like a dictionary with Boolean elements. >> > >> > There's even a combinator all/1 so that >> > All = lens:c(lens:tuple(1)), >> > Data = [{3,a},{1,b},{4,c},{6,d}] >> > lens:get(All, Data) >> > => [3,1,4,6] >> > lens:update(All, Data, fun (N) -> N*10 end) >> > => [{30,a},{10,b},{40,c},{60,d}] >> > >> > So what about cross-module inlining? >> > >> > By the time you've made and composed a couple of lenses, >> > you have a tangle of funs. If the compiler (a) always >> > inlines (fun (...) -> ... end)(...) and (b) inlines the >> > functions from the lens module, then the tangle resolves >> > tidily into reasonably straightforward code. >> > >> > But the cross-module inlining step is crucial. Without >> > that, the compiler has nothing to work on. >> > >> > At its crudest, consider >> > >> > lens:get(lens:tuple(#rec.fld), Rec) >> > >> > *With* cross-module inlining and inlining of funs that >> > are applied to known arguments and dead code elimination, >> > you end up with >> > >> > element(#rec.fld, Rec) >> > >> > *Without* cross-module inlining, three funs are built, >> > only one of which will ever be called. >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > J. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Nov 26 17:36:06 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 26 Nov 2015 17:36:06 +0100 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Message-ID: On Thu, Nov 26, 2015 at 5:00 PM, Dan Gudmundsson wrote: > Maybe a stupid question I have not looked at lenses before > but why include the "path of keys" into the lens? My guess is that requiring a Key breaks some trivial lenses, or adds some complexity to the proofs. You can create an identity lens for instance: lens_id() -> {fun(X) -> X end, fun(A, _X) -> A, ...}. which should obey the lens laws. Furthermore it composes as an identity does, i.e. compose(ID, Lens) == Lens compose(Lens, ID) == Lens It could be, I haven't really checked, that the requirement of a Key poses additional complexity here. Lenses stems from languages which have currying, so you can simply partially apply the key to obtain the closure which can process that Key. By doing so, the Key kind-of gets out of the way, which probably makes identities as the above fall out nicer in the logic, and currying can be used to elegantly "hide" the fact that there are keys in there over which we have built a closure. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Nov 27 04:42:51 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 27 Nov 2015 16:42:51 +1300 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Message-ID: On 27/11/2015, at 2:35 am, Tristan Sloughter wrote: > Jesper also created https://github.com/jlouis/erl-lenses a few years > back for this. Somehow I had managed to forget that. My sincere apologies. Since we've just had a thread about updates, again, one of many since he wrote that, it's certainly time the answer was better known. For what it's worth, my lens.erl fixes a performance problem that his lens.erl still has And I was very pleased with myself for thinking up the fix until I discovered that it was thoroughly familiar in the Haskell community. > https://github.com/jlouis/erl-lenses defines a lens as a pair {Get,Put}. http://www.cs.otago.ac.nz/staffpriv/ok/lens.erl defines as lens as a triple {Get,Put,Update}. Now *semantically* you should be able to replace update(Lens, Data, Fun) by put(Lens, Data, Fun(get(Lens, Data)). and indeed my lens.erl provides a function to fill in an Update given a Get and Put. But they are *pragmatically* not the same. Let's take the simple case of a list of Numbers, and adding 1 to the Nth of them. Lens = index(N), put(Lens, List, 1 + Get(Lens, List)) traverses the first N elements of the list *twice*, while Lens = index(N), update(Lens, List, fun (X) -> X + 1 end) traverses the first N elements of the list *once*. From an actual measurement, put+get, full price : 100% update, full price : 50% put+get, inlined : 65% update, inlined : 45% The inlining was done by hand. Doing it in a single pass also applies to tree-like data. (I'll make another post suggesting an addition to gb_trees.erl.) (A word of warning: this is an example where the cost of doing the update dwarfs the cost of going through a lens. Don't expect the ratios to be so close for record fields.) The really nice thing is that the update functions are even easier to compose than the put functions. I should have mentioned another law: Put(X, Y) = Update(X, fun (_) -> Y end). Pierre Fenoll pierrefenoll@REDACTED said > That's something the JIT should be doing, right? Answer: cross-module inlining *changes the semantics* of Erlang. So with the exception of a handful of core system modules, I would hope that HyPE *doesn't* do that. Cross-module inlining should be licensed by an explicit declaration of some kind. Back in the last century I proposed an -import_static(Module, [...Function...]). directive to tell the compiler to bind the source module to the version of Module available at that time to the compiler (and to check at load time that the same version was used). > What happens when inlined code throws an error? > Which location information is included? I wouldn't expect cross-module inlining to be any different from the inlining that's done now. The point of inlining is to integrate the called function into the calling function. The same exceptions would be raised, but appearing to come from the caller, not the callee. Since the actual code error would (almost always) be in the caller, that's the place we want to look anyway. From ok@REDACTED Fri Nov 27 05:28:14 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 27 Nov 2015 17:28:14 +1300 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Message-ID: <26BE86AA-0B6D-419B-9EB1-833C840ABFFE@cs.otago.ac.nz> On 27/11/2015, at 5:00 am, Dan Gudmundsson wrote: > Nice > > Maybe a stupid question I have not looked at lenses before > but why include the "path of keys" into the lens? It's not at all a stupid question. At least, since I pondered the same thing myself, I *hope* it wasn't a stupid question. > Why not make them reusable and define > get({G,_,_}, Key, X) -> G(Key, X). > put({_,P,_}, Key, X, Y) -> P(Key, X, Y). > update({_,_,U}, Key, X, F) -> U(Key, X, F). There are several answers. (1) There are lenses like lens:hd() that don't *have* an argument. While [get/3,put/4,update/4] *could* exist, [get/2,put/3,update/3] *must* exist in order to handle those. (2) There are also lenses like lens:c(lens:index(I), lens:index(J)) -- an index into a 2D array represented as a list of lists which have *two* arguments. So now we're up to get/2, put/3, update/3, get/3, put/4, update/4, get/4, put/5, update/5 and where do we stop? (3) In terms of *functional sufficiency*, the present scheme is enough. If we want tuple() -> fun (Index) -> lens:tuple(Index) end we can write it. So the issue is performance. (4) If lenses were implemented by a parse transform or by cross-module inlining, there wouldn't *be* any performance advantage for get/3, put/4, update/4. While they aren't, I'm more troubled by the cost of building three closures when only one will be called. (5) Remember the idea, to make "selectors" first class values? Splitting a selector into how-to-select + which-to-select pretty much destroys that. Here's a C++ analogy, if it's any use. Think of a Lens as being like a pointer-to-member, which in C++ is a thing that doesn't point to a member of a specific object (that's a plain pointer) but to a pointer to a specific *part* of any object (of the right type) it happens to be applied to. (6) Finally, I for one am a bear of very little brain. It took me a while to get my head around lenses enough to realise that they *could* work for Erlang; the simpler the scheme remains, the more likely it is that I'll be able to use it. > > Another question, should/could we add a fourth fun (map) for traversal? No. > As it is all/1 and where/1 assumes lists and does not operate on for example gb_trees or dict. Yes, but all/1 and where/1 are functions that make SPECIALISED lenses for SPECIALISED data. Lenses are primarily a way of solving the update-a-value-in-a-deep-nest-of-records problem; there is nothing about Lenses *as such* that has anything to do with containers. all/1 was just included to show that the data you are extracting need not exist as a single value in the thing you're extracting the data from. (Like you can extract a column from a matrix that is stored by rows.) Here's another: integer() -> { fun (N) -> integer_to_list(N) end , fun (_, L) -> list_to_integer(L) end , fun (N, F) -> list_to_integer(F(integer_to_list(N))) end }. 2> lens:get(lens:integer(), 42). "42" 3> lens:put(lens:integer(), 42, "35"). 35 4> lens:update(lens:integer(), 42, fun (L) -> L++"13" end). 4213 5> lens:put(lens:integer(), 'FOO!', "65"). 65 In all honesty, I have no idea what a Map function would look like in a Lens. If you have any idea of what Map would mean for a record field, more power to you. What I *have* wondered about is information that might or might not exist. Should it be done with special functions in every lens, or should it be done by having special lenses? % gb_lookup(K) -> lens(gb_tree(K,V), none | {value,V}). gb_lookup(Key) -> { fun (Tree) -> gb_trees:lookup(Key, Tree) end , fun (Tree, none) -> gb_trees:delete_any(Key, Tree) ; (Tree, {value,V}) -> gb_trees:enter(Key, V, Tree) end , fun (Tree, F) -> case gb_trees:lookup(Key, Tree) of none -> Tree ; {value,V} -> gb_trees:update(Key, F(V), Tree) end end }. The sticking point was list and tuple indexing, where it does make sense to have an optional fetch, but not a store which can delete. So maybe an opt(Lens, Data, Default) might make sense, but not a put or update analogue. From ok@REDACTED Fri Nov 27 05:41:16 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 27 Nov 2015 17:41:16 +1300 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Message-ID: <42F4BB99-7837-4F5B-AD6B-DCBFF7A6E270@cs.otago.ac.nz> On 27/11/2015, at 5:36 am, Jesper Louis Andersen wrote: You can create an identity lens for instance: > > lens_id() -> > {fun(X) -> X end, fun(A, _X) -> A, ...}. I've just added that, with credit. From mark@REDACTED Fri Nov 27 03:20:43 2015 From: mark@REDACTED (Mark Steele) Date: Thu, 26 Nov 2015 21:20:43 -0500 Subject: [erlang-questions] where is os:getuid() ? Message-ID: Happened across a post to the mailing list regarding getuid. Just added it here: https://github.com/marksteele/passwderl Add it to your rebar config, and you're good to go. Cheers, Mark Steele CISSP, GPEN, GCIA, CSM (647) 677-4064 mark@REDACTED LinkedIn: https://ca.linkedin.com/in/markrsteele Github: https://github.com/marksteele Personal: http://www.control-alt-del.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Nov 27 06:14:49 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 27 Nov 2015 18:14:49 +1300 Subject: [erlang-questions] Proposed addition to gb_trees Message-ID: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> I would like to propose adding the following function to gb_trees, and similar functions to other dictionary-like things that don't have one. -spec updatef(term(), fun((term()) -> term()), gb_tree()) -> gb_tree(). updatef(Key, Fun, {S, T}) -> T1 = updatef_1(Key, Fun, T), {S, T1}. updatef_1(Key, Fun, {Key1, V, Smaller, Bigger}) when Key < Key1 -> {Key1, V, updatef_1(Key, Fun, Smaller), Bigger}; updatef_1(Key, Fun, {Key1, V, Smaller, Bigger}) when Key > Key1 -> {Key1, V, Smaller, updatef_1(Key, Fun, Bigger)}; updatef_1(Key, Fun, {_, V, Smaller, Bigger}) -> {Key, Fun(V), Smaller, Bigger}. The idea is that updatef(Key, Fun, Tree) = update(Key, Fun(get(Key, Tree)), Tree) except that it makes a single pass through the tree, not two passes. There are two questions about this, which is why I haven't submitted patches. (1) What's a good name for the function? Update really is the right name for it, but the existing 'replace' function is called 'update'. (2) What should happen if the key is not present in the tree? gb_trees as its stands has an interface I find complex because practically everything occurs in two or three copies: assume key present, assume key absent, allow for either possibility. Assuming it's absent doesn't make sense here, because there'd be nothing to pass to Fun. But that leaves two copies, one which would err if the key was absent and the other which would just not change the tree. From mjtruog@REDACTED Fri Nov 27 06:30:48 2015 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 26 Nov 2015 21:30:48 -0800 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Message-ID: <5657EA88.1010901@gmail.com> On 11/26/2015 07:42 PM, Richard A. O'Keefe wrote: > On 27/11/2015, at 2:35 am, Tristan Sloughter wrote: > >> That's something the JIT should be doing, right? > Answer: cross-module inlining *changes the semantics* > of Erlang. So with the exception of a handful of core > system modules, I would hope that HyPE *doesn't* do that. > > Cross-module inlining should be licensed by an explicit > declaration of some kind. Back in the last century I > proposed an > -import_static(Module, [...Function...]). > directive to tell the compiler to bind the source module > to the version of Module available at that time to the > compiler (and to check at load time that the same version > was used). Having it be opt-in does sound good, but it should complicate hot-code loading, making a module update possibly fail due to the import_static dependency on an external module. If module 'a' uses import_static on module 'b', does the inlining (actual source code) of module 'a' change during runtime if module 'b' changes? When the module 'b' update indirectly alters module 'a', does its module version not change? If you were allowed to do an update of a module used for inlining, it could create a ripple-effect in the system, making the process more brittle and prone to failure, due to the lack of isolation. If instead the modules that are inlined are marked "permanent" and prevented from being updated in the future, then there is a reason to not use inlining (which becomes problematic, since it is only controlled in other modules that may be beyond the scope of your development concerns (e.g., a lower-level library that you want to be able to update in the future)). You may be able to make a lens implementation that relies on header file usage, which would allow all the functions to be within a single module and allowed to be inlined there. That approach wouldn't cause problems for module updating in the future. From denc716@REDACTED Fri Nov 27 09:46:21 2015 From: denc716@REDACTED (derek) Date: Fri, 27 Nov 2015 00:46:21 -0800 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: Message-ID: On Thu, Nov 26, 2015 at 6:20 PM, Mark Steele wrote: > Happened across a post to the mailing list regarding getuid. > > Just added it here: > > https://github.com/marksteele/passwderl Thanks and that's nice to have! While has anyone tried to make it available into Erlang os module ? once we have os:getuid/0, then someone might want chown, chmod, ...; Even though windows don't have it, why not make it OS specific like the Python os module? https://docs.python.org/3/library/os.html#os.chown Availability: Unix. > > Add it to your rebar config, and you're good to go. > > Cheers, > > > Mark Steele > CISSP, GPEN, GCIA, CSM > (647) 677-4064 > mark@REDACTED > > LinkedIn: https://ca.linkedin.com/in/markrsteele > Github: https://github.com/marksteele > Personal: http://www.control-alt-del.org From dangud@REDACTED Fri Nov 27 10:29:27 2015 From: dangud@REDACTED (Dan Gudmundsson) Date: Fri, 27 Nov 2015 09:29:27 +0000 Subject: [erlang-questions] Proposed addition to gb_trees In-Reply-To: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> References: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> Message-ID: On Fri, Nov 27, 2015 at 6:21 AM Richard A. O'Keefe wrote: > (1) What's a good name for the function? Update really is > the right name for it, but the existing 'replace' > function is called 'update'. > Sigh, maybe 'modify' or 'change'. > (2) What should happen if the key is not present in the > tree? gb_trees as its stands has an interface I > find complex because practically everything occurs > in two or three copies: assume key present, assume key > absent, allow for either possibility. Assuming it's > absent doesn't make sense here, because there'd be > nothing to pass to Fun. But that leaves two copies, > one which would err if the key was absent and the other > which would just not change the tree. > IMO it should behave as gb_trees:update/3, dict, orddict and maps does, i.e. assume the key is present, you can not update something that is not there. Add docs and test cases to the pullreq. And while you are at it the array module need an update/3 as well :-) > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Fri Nov 27 12:22:38 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 27 Nov 2015 12:22:38 +0100 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> Message-ID: On Fri, Nov 27, 2015 at 4:42 AM, Richard A. O'Keefe wrote: > defines as lens as a triple {Get,Put,Update}. This! Richard's implementation is better for numerous reasons, among which is that it provides an optimized Update-operation. When developing the logic, it is nice to say "Update is really just an application of the identity put(Lens, Data, Fun(get(Lens, Data))" because those kind of observations tend to shorten proofs. In practice however, there are often a speedup to be gained by recognizing code often has a nice way to "update-in-place" when a target element is found. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Fri Nov 27 13:52:22 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 27 Nov 2015 13:52:22 +0100 Subject: [erlang-questions] Proposed addition to gb_trees In-Reply-To: References: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> Message-ID: On Fri, Nov 27, 2015 at 10:29 AM, Dan Gudmundsson wrote: > IMO it should behave as gb_trees:update/3, dict, orddict and maps does, > i.e. assume the key is present, you can not update something that is not > there. > The alternative is a function of the form (undefined | {value, Key, Value}) -> undefined | {ok, Value} | {error, Reason}. This allows the function to decide and in particular implement "upsert"-like functionality. This is not too far-fetched: OCaml's Core library calls the updater 'change' in this case. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zandra.hird@REDACTED Fri Nov 27 14:34:17 2015 From: zandra.hird@REDACTED (Zandra Hird) Date: Fri, 27 Nov 2015 14:34:17 +0100 Subject: [erlang-questions] Patch Package OTP 18.1.5 Released Message-ID: <56585BD9.2090506@ericsson.com> Patch Package: OTP 18.1.5 Git Tag: OTP-18.1.5 Date: 2015-11-27 Trouble Report Id: OTP-13140 Seq num: System: OTP Release: 18 Application: ssh-4.1.3 Predecessor: OTP 18.1.4 Check out the git tag OTP-18.1.5, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- ssh-4.1.3 ------------------------------------------------------- --------------------------------------------------------------------- The ssh-4.1.3 application can be applied independently of other applications on a full OTP 18 installation. --- Known Bugs and Problems --- OTP-13140 Application(s): ssh SSH_MSG_KEX_DH_GEX_REQUEST_OLD implemented to make PuTTY work with erl server. Full runtime dependencies of ssh-4.1.3: crypto-3.3, erts-6.0, kernel-3.0, public_key-0.22, stdlib-2.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From michael.santos@REDACTED Fri Nov 27 17:08:18 2015 From: michael.santos@REDACTED (Michael Santos) Date: Fri, 27 Nov 2015 11:08:18 -0500 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: Message-ID: <20151127160818.GA26354@brk> On Thu, Nov 26, 2015 at 09:20:43PM -0500, Mark Steele wrote: > Happened across a post to the mailing list regarding getuid. > > Just added it here: > > https://github.com/marksteele/passwderl > > Add it to your rebar config, and you're good to go. A few comments: * getuid(2), seteuid(2), etc take and return uid_t's and gid_t's. These are unsigned 32-bit integers on the platforms I checked, not signed values. In particular, passing a signed value into seteuid(2) may have unexpected results. * the getpw* calls are blocking and may take some time to complete since names may be resolved against network services like NIS or LDAP. * there are problems portably using setuid(2) and seteuid(2) that were documented in "Setuid Demystified": https://www.cs.berkeley.edu/~daw/papers/setuid-usenix02.pdf A more predictable interface is setresuid(2) which seems to be available on all Unixes except Solaris. > Cheers, > > > Mark Steele > CISSP, GPEN, GCIA, CSM > (647) 677-4064 > mark@REDACTED > > LinkedIn: https://ca.linkedin.com/in/markrsteele > Github: https://github.com/marksteele > Personal: http://www.control-alt-del.org > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From michael.santos@REDACTED Fri Nov 27 17:10:48 2015 From: michael.santos@REDACTED (Michael Santos) Date: Fri, 27 Nov 2015 11:10:48 -0500 Subject: [erlang-questions] where is os:getuid() ? In-Reply-To: References: Message-ID: <20151127161048.GB26354@brk> On Fri, Nov 27, 2015 at 12:46:21AM -0800, derek wrote: > On Thu, Nov 26, 2015 at 6:20 PM, Mark Steele wrote: > > Happened across a post to the mailing list regarding getuid. > > > > Just added it here: > > > > https://github.com/marksteele/passwderl > > Thanks and that's nice to have! > > While has anyone tried to make it available into Erlang os module ? > once we have os:getuid/0, then someone might want chown, chmod, ...; These are in the file module: file:change_owner/2,3, file:write_file_info/2,3 > Even though windows don't have it, why not make it OS specific like > the Python os module? > > https://docs.python.org/3/library/os.html#os.chown > > Availability: Unix. > > > > > Add it to your rebar config, and you're good to go. > > > > Cheers, > > > > > > Mark Steele > > CISSP, GPEN, GCIA, CSM > > (647) 677-4064 > > mark@REDACTED > > > > LinkedIn: https://ca.linkedin.com/in/markrsteele > > Github: https://github.com/marksteele > > Personal: http://www.control-alt-del.org > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Fri Nov 27 17:27:47 2015 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 27 Nov 2015 08:27:47 -0800 Subject: [erlang-questions] Proposed addition to gb_trees In-Reply-To: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> References: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> Message-ID: <56588483.6030909@gmail.com> On 11/26/2015 09:14 PM, Richard A. O'Keefe wrote: > (2) What should happen if the key is not present in the > tree? gb_trees as its stands has an interface I > find complex because practically everything occurs > in two or three copies: assume key present, assume key > absent, allow for either possibility. Assuming it's > absent doesn't make sense here, because there'd be > nothing to pass to Fun. But that leaves two copies, > one which would err if the key was absent and the other > which would just not change the tree. The dict interface has an update with arity 4 to allow a default value to be provided. The same approach could be taken here, where the arity 3 update throws an exception if the key does not exist. From jean.parpaillon@REDACTED Fri Nov 27 17:28:36 2015 From: jean.parpaillon@REDACTED (Jean Parpaillon) Date: Fri, 27 Nov 2015 17:28:36 +0100 Subject: [erlang-questions] erlang D-Bus implementation ? Message-ID: <1448641716.4193.81.camel@free.fr> Hi all, There have been several tentative to have a pure erlang implementation of D-Bus RPC system, most of them forks of (unmaintained)?http://source forge.net/projects/dbus-erlang/ I have also forked it some years ago and use it in another project. I can't say it's perfectly operational but at least for consuming D-Bus services it works. Exposing D-Bus services is not possible. The original erlang-dbus was proposing a gen_dbus behaviour for that but I preferred to focus on robust service consuming first. My version is there:?https://github.com/lizenn/erlang-dbus/ TODO list: - documentation ;) - testing: test infra is operational (with travis-ci) but coverage is really low - gen_dbus coding - well know interfaces coding: Properties (partly coded in dbus_properties_proxy), ObjectManager, ... In a far future, we could imagine OTP to include it, as DBus is used for many services and as cros-language RPC system. The only problem regarding OTP is most DBus implementations runs on top of UNIX sockets, which requires platform dependant code. OTP version could be operational with TCP sockets (implemented in erlang-dbus), with an option of communicating through UNIX socket with ports. Is anybody interested in contributing ? -- Jean Parpaillon -- Open Source Consultant OW2 Technology Council Chairman Director @ OW2 Consortium OCCIware Strategic Orientation Committee Chairman Research Engineer @ Inria -- Phone: +33 6 30 10 92 86 im: jean.parpaillon@REDACTED skype: jean.parpaillon linkedin: http://www.linkedin.com/in/jeanparpaillon/en From mjtruog@REDACTED Fri Nov 27 18:31:00 2015 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 27 Nov 2015 09:31:00 -0800 Subject: [erlang-questions] erlang D-Bus implementation ? In-Reply-To: <1448641716.4193.81.camel@free.fr> References: <1448641716.4193.81.camel@free.fr> Message-ID: <56589354.60304@gmail.com> On 11/27/2015 08:28 AM, Jean Parpaillon wrote: > Hi all, > There have been several tentative to have a pure erlang implementation > of D-Bus RPC system, most of them forks of (unmaintained) http://source > forge.net/projects/dbus-erlang/ > > I have also forked it some years ago and use it in another project. I > can't say it's perfectly operational but at least for consuming D-Bus > services it works. Exposing D-Bus services is not possible. The > original erlang-dbus was proposing a gen_dbus behaviour for that but I > preferred to focus on robust service consuming first. > > My version is there: https://github.com/lizenn/erlang-dbus/ > > TODO list: > - documentation ;) > - testing: test infra is operational (with travis-ci) but coverage is > really low > - gen_dbus coding > - well know interfaces coding: Properties (partly coded in > dbus_properties_proxy), ObjectManager, ... > > In a far future, we could imagine OTP to include it, as DBus is used > for many services and as cros-language RPC system. The only problem > regarding OTP is most DBus implementations runs on top of UNIX sockets, > which requires platform dependant code. OTP version could be > operational with TCP sockets (implemented in erlang-dbus), with an > option of communicating through UNIX socket with ports. > > Is anybody interested in contributing ? You can track UNIX socket support at https://github.com/erlang/otp/pull/612 . From olopierpa@REDACTED Fri Nov 27 18:55:35 2015 From: olopierpa@REDACTED (Pierpaolo Bernardi) Date: Fri, 27 Nov 2015 18:55:35 +0100 Subject: [erlang-questions] Proposed addition to gb_trees In-Reply-To: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> References: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> Message-ID: On Fri, Nov 27, 2015 at 6:14 AM, Richard A. O'Keefe wrote: > (2) What should happen if the key is not present in the > tree? gb_trees as its stands has an interface I > find complex because practically everything occurs > in two or three copies: assume key present, assume key > absent, allow for either possibility. Assuming it's > absent doesn't make sense here, because there'd be > nothing to pass to Fun. But that leaves two copies, > one which would err if the key was absent and the other > which would just not change the tree. It should take an optional initializer parameter, like for example the corresponding function in Racket. http://docs.racket-lang.org/reference/hashtables.html#%28def._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._hash-update%29%29 From technion@REDACTED Sat Nov 28 12:26:55 2015 From: technion@REDACTED (Technion) Date: Sat, 28 Nov 2015 11:26:55 +0000 Subject: [erlang-questions] ssl:connect fail Message-ID: Hi, I was wondering if I could get some assistance with an application I'm building. Up until now, I've been running development starting it with: ./rebar3 shell This works (although the app isn't finished). It starts my gen_server automatically and I can sit there watching the debug logging run (which seems to be more than the documented "Run shell with project apps in path"). I also have a series of eunit tests that run fine, and xref doesn't complain. However, attempting to turn this into a release and start it seems to fail: $ ./rebar3 release $ /home/technion/ct_advisor/_build/default/rel/ct_advisor/bin/ct_advisor console Exec: /usr/lib/erlang/erts-7.1/bin/erlexec -boot /home/technion/ct_advisor/_build/default/rel/ct_advisor/releases/0.1.0/ct_advisor -boot_var ERTS_LIB_DIR /usr/lib/erlang/erts-7.1/../lib -config /home/technion/ct_advisor/_build/default/rel/ct_advisor/releases/0.1.0/sys.config -args_file /home/technion/ct_advisor/_build/default/rel/ct_advisor/releases/0.1.0/vm.args -- console Root: /home/technion/ct_advisor/_build/default/rel/ct_advisor /home/technion/ct_advisor/_build/default/rel/ct_advisor Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] [async-threads:30] [hipe] [kernel-poll:true] 11:15:08.781 [info] Application lager started on node ct_advisor@REDACTED 11:15:08.799 [error] gen_server <0.79.0> terminated with reason: {'module could not be loaded',[{ssl,connect,["ct.googleapis.com",443,[{nodelay,true},binary,{active,false}],30000],[]},{ibrowse_http_client,send_req_1,8,[{file,"/home/technion/ct_advisor/_build/default/lib/ibrowse/src/ibrowse_http_client.erl"},{line,758}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,629}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,661}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} ... lots of stuff snipped. The failure here appears to be within ibrowse, so I'm expecting it's not a code error. It also appears to feel ssl:connect doesn't load. There are lots of Google hits for this type of error, suggesting Erlang wasn't built with SSL support. But I have eunit tests that actually run ibrowse (to https: addresses) and they work. And the whole thing runs when run from within a shell. I'm totally lost as to why a release (which seems to build fine) would be magically different. Version information: $ ./rebar3 -v rebar 3.0.0-beta.4+build.188.ref73c3c55 on Erlang/OTP 18 Erts 7.1 Any assistance appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sat Nov 28 18:51:57 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Sat, 28 Nov 2015 18:51:57 +0100 Subject: [erlang-questions] ssl:connect fail In-Reply-To: Message-ID: <25f32c36-0ffc-4214-8b00-10d88e57b85d@email.android.com> Hi, Remind that a release come with the only needed applications and modules, while shell can have access to the whole Erlang applications and modules. But it is not magic. Your .app file must declare things. See 'app' in Erlang documentation. 'application' key must declare all modules and applications needed by your own application, in order to relx build a valid start script. Regards Le?28 nov. 2015 12:26 PM, Technion a ?crit?: > > Hi, > > > I was wondering if I could get some assistance with an application I'm building. > > > Up until now, I've been running development starting it with: > > > ./rebar3 shell > > > This works (although the app isn't finished). It starts my gen_server automatically and I can sit there watching the debug logging run (which seems to be more than the documented "Run shell with project apps in path"). I also have a series of eunit tests that run fine, and xref doesn't complain. > > > However, attempting to turn this into a release and start it seems to fail: > > > $ ./rebar3 release > > $ /home/technion/ct_advisor/_build/default/rel/ct_advisor/bin/ct_advisor console > Exec: /usr/lib/erlang/erts-7.1/bin/erlexec -boot /home/technion/ct_advisor/_build/default/rel/ct_advisor/releases/0.1.0/ct_advisor -boot_var ERTS_LIB_DIR /usr/lib/erlang/erts-7.1/../lib -config /home/technion/ct_advisor/_build/default/rel/ct_advisor/releases/0.1.0/sys.config -args_file /home/technion/ct_advisor/_build/default/rel/ct_advisor/releases/0.1.0/vm.args -- console > Root: /home/technion/ct_advisor/_build/default/rel/ct_advisor > /home/technion/ct_advisor/_build/default/rel/ct_advisor > Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] [async-threads:30] [hipe] [kernel-poll:true] > > 11:15:08.781 [info] Application lager started on node ct_advisor@REDACTED > 11:15:08.799 [error] gen_server <0.79.0> terminated with reason: {'module could not be loaded',[{ssl,connect,["ct.googleapis.com",443,[{nodelay,true},binary,{active,false}],30000],[]},{ibrowse_http_client,send_req_1,8,[{file,"/home/technion/ct_advisor/_build/default/lib/ibrowse/src/ibrowse_http_client.erl"},{line,758}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,629}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,661}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} > > ... lots of stuff snipped. > > > > The failure here appears to be within ibrowse, so I'm expecting it's not a code error. It also appears to feel ssl:connect doesn't load. > > > There are lots of Google hits for this type of error, suggesting Erlang wasn't built with SSL support. But I have eunit tests that actually run ibrowse (to https: addresses) and they work. And the whole thing runs when run from within a shell. > > > I'm totally lost as to why a release (which seems to build fine) would be magically different. > > > Version information: > > $ ./rebar3 -v > rebar 3.0.0-beta.4+build.188.ref73c3c55 on Erlang/OTP 18 Erts 7.1 > > > Any assistance appreciated. > From mononcqc@REDACTED Sat Nov 28 20:15:23 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Sat, 28 Nov 2015 14:15:23 -0500 Subject: [erlang-questions] ssl:connect fail In-Reply-To: <25f32c36-0ffc-4214-8b00-10d88e57b85d@email.android.com> References: <25f32c36-0ffc-4214-8b00-10d88e57b85d@email.android.com> Message-ID: <20151128191521.GA877@ferdmbp.local> On 11/28, ?ric Pailleau wrote: >Hi, >Remind that a release come with the only needed applications and modules, while shell can have access to the whole Erlang applications and modules. >But it is not magic. Your .app file must declare things. >See 'app' in Erlang documentation. >'application' key must declare all modules and applications needed by your own application, in order to relx build a valid start script. > >Regards > To expand on this, make sure your application has the 'ssl' application in its `{application, [...]}' tuple in its .app (or .app.src) file. This will let relx know to pull it into the final release properly. Another alternative is to jut put the 'ssl' app in the relx configuration, but this is a weaker form of it: while it will pull the app properly into the release, it won't ensure it is booted before your app does; the .app data is what conveys that information. From technion@REDACTED Sat Nov 28 22:25:47 2015 From: technion@REDACTED (Technion) Date: Sat, 28 Nov 2015 21:25:47 +0000 Subject: [erlang-questions] ssl:connect fail In-Reply-To: <20151128191521.GA877@ferdmbp.local> References: <25f32c36-0ffc-4214-8b00-10d88e57b85d@email.android.com>, <20151128191521.GA877@ferdmbp.local> Message-ID: Many thanks, that seemed to do it. Of course I have new problems now :) ________________________________________ From: Fred Hebert Sent: Sunday, 29 November 2015 6:15 AM To: ?ric Pailleau Cc: Technion; Erlang Questions Subject: Re: [erlang-questions] ssl:connect fail On 11/28, ?ric Pailleau wrote: >Hi, >Remind that a release come with the only needed applications and modules, while shell can have access to the whole Erlang applications and modules. >But it is not magic. Your .app file must declare things. >See 'app' in Erlang documentation. >'application' key must declare all modules and applications needed by your own application, in order to relx build a valid start script. > >Regards > To expand on this, make sure your application has the 'ssl' application in its `{application, [...]}' tuple in its .app (or .app.src) file. This will let relx know to pull it into the final release properly. Another alternative is to jut put the 'ssl' app in the relx configuration, but this is a weaker form of it: while it will pull the app properly into the release, it won't ensure it is booted before your app does; the .app data is what conveys that information. From schneider@REDACTED Sun Nov 29 14:27:27 2015 From: schneider@REDACTED (Frans Schneider) Date: Sun, 29 Nov 2015 14:27:27 +0100 Subject: [erlang-questions] tcp data in a gen_fsm Message-ID: <565AFD3F.9080508@xs4all.nl> Dear list, I have a gen_fsm representing the tcp connection to a remote server with 6 states defined. Tcp data is handled by handle_info with active = once.Tcp packets can contain more than one commands/data items. I would like to feed the data into the different FSM states in the same manner as with send_event() and not handle the different states in separate handle_info clauses for the different states or a massive handle_info clause with a huge case statement. Question is, can I call the state clauses like this or do I brake OTP behavior?This is a client and blocking incoming data is not an issue. handle_info({tcp, Socket, Raw_data}, State_name, #state{buffer = Buffer} = State) -> {Datas, Buffer1} = unpack(<>), lists:foldl(fun(Data, {StateName_in, State_in}) -> {next_state, StateName_out, State_out} = ?MODULE:StateName_in({recv, Data}, State_in), {StateName_out, State_out} end, {State_name, State}, Datas), inet:setopts(Socket, [{active, once}]), {next_state, active, State#state{buffer = Buffer1}}; I can of course use a seperate process for handling the tcp stuff which makes the gen_fsm calls, but was just wondering if this could work as well. Thanks, /Frans From schneider@REDACTED Sun Nov 29 14:39:12 2015 From: schneider@REDACTED (Frans Schneider) Date: Sun, 29 Nov 2015 14:39:12 +0100 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: <565AFD3F.9080508@xs4all.nl> References: <565AFD3F.9080508@xs4all.nl> Message-ID: <565B0000.7010602@xs4all.nl> I forgot to fetch the last state name and state data from the lists:foldl() as return values of the handle_info() in the code below. On 11/29/2015 02:27 PM, Frans Schneider wrote: > Dear list, > > I have a gen_fsm representing the tcp connection to a remote server > with 6 states defined. Tcp data is handled by handle_info with active > = once.Tcp packets can contain more than one commands/data items. > I would like to feed the data into the different FSM states in the > same manner as with send_event() and not handle the different states > in separate handle_info clauses for the different states or a massive > handle_info clause with a huge case statement. > Question is, can I call the state clauses like this or do I brake OTP > behavior?This is a client and blocking incoming data is not an issue. > > handle_info({tcp, Socket, Raw_data}, State_name, #state{buffer = > Buffer} = State) -> > {Datas, Buffer1} = unpack(<>), > lists:foldl(fun(Data, {StateName_in, State_in}) -> > {next_state, StateName_out, State_out} = > ?MODULE:StateName_in({recv, Data}, State_in), > {StateName_out, State_out} > end, {State_name, State}, Datas), > inet:setopts(Socket, [{active, once}]), > {next_state, active, State#state{buffer = Buffer1}}; > > I can of course use a seperate process for handling the tcp stuff > which makes the gen_fsm calls, but was just wondering if this could > work as well. > > Thanks, > > /Frans The code now is: handle_info({tcp, Socket, Raw_data}, State_name, #state{buffer = Buffer} = State) -> {Datas, Buffer1} = unpack(<>), {State_name1, State1} = lists:foldl( fun(Data, {StateName_in, State_in}) -> {next_state, StateName_out, State_out} = ?MODULE:StateName_in({recv, Data}, State_in), {StateName_out, State_out} end, {State_name, State}, Datas), inet:setopts(Socket, [{active, once}]), {next_state, State_name1, State1#state{buffer = Buffer1}}; From zxq9@REDACTED Sun Nov 29 14:54:28 2015 From: zxq9@REDACTED (zxq9) Date: Sun, 29 Nov 2015 22:54:28 +0900 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: <565AFD3F.9080508@xs4all.nl> References: <565AFD3F.9080508@xs4all.nl> Message-ID: <1516474.15cRhksAXU@changa> On 2015?11?29? ??? 14:27:27 Frans Schneider wrote: > Dear list, > > I have a gen_fsm representing the tcp connection to a remote server with > 6 states defined. Tcp data is handled by handle_info with active = > once.Tcp packets can contain more than one commands/data items. > I would like to feed the data into the different FSM states in the same > manner as with send_event() and not handle the different states in > separate handle_info clauses for the different states or a massive > handle_info clause with a huge case statement. > Question is, can I call the state clauses like this or do I brake OTP > behavior?This is a client and blocking incoming data is not an issue. > > handle_info({tcp, Socket, Raw_data}, State_name, #state{buffer = Buffer} > = State) -> > {Datas, Buffer1} = unpack(<>), > lists:foldl(fun(Data, {StateName_in, State_in}) -> > {next_state, StateName_out, State_out} = > ?MODULE:StateName_in({recv, Data}, State_in), > {StateName_out, State_out} > end, {State_name, State}, Datas), > inet:setopts(Socket, [{active, once}]), > {next_state, active, State#state{buffer = Buffer1}}; > > I can of course use a seperate process for handling the tcp stuff which > makes the gen_fsm calls, but was just wondering if this could work as well. Using FSMs to represent protocol state is usually a win. However, using it to deal with the socket directly in TCP usually isn't a good fit. TCP is a stream, and you understand how that works as evidenced by your unpack/1 function. That means your variable Datas should contain a list of received messages in your protocol -- and each message has the potential to change the state of the protocol. You are now going to be looping over passed FSM states in *two* cases: when you receive new network data, and when you have anything left in the message list held in Datas. That's a little awkward, and defeats the purpose of passing NextState to the FSM -- since you have to manually handle the difference in your looping code. I prefer instead to write a TCP process that deals only with network messages and does something like unpack/1 on them (but I have a sort of generic TCP process I use for this, where unpack/1 is a callback -- and I've always wondered why OTP doesn't have something like this, since gen_servers are just a touch kludgy for this ...?). Any time Datas is not empty the TCP process sends each element of Datas, one by one as Erlang messages to the FSM that represents the protocol state. TCP is its own protocol. It needs a handler. Your protocol is its own thing as well, and an FSM suits that well. The two protocols being written into a single process gets more convoluted than I enjoy. ...but... ZOMG MY PRECIOUS PERFORMANSSESSSS!!! And, well, whatever. I've never had *this* part of a program be my performance bottleneck. I have had this part of a program cause me to waste a lot of *my* time picking out the difference between unpacking TCP, iterating over a stack of extracted protocol messages, and iterating over the network receives. It certainly *can* be made to work, and it appears that *most* code is written like this -- where the application and transmission stuff is mixed together -- but I don't find it to be particularly readable a few months later, and that sucks really bad when you're trying to figure out what your protocol is up to. Its also *very* nice to separate your protocol out into Erlang messages (even if its a binary or text protocol and not based on Erlang terms) because if you separate the network bits from the protocol bits you can use your own application protocol within your own node to do other stuff -- especially testing of your process' compliance with your own protocol. It also gives you a lot of component flexibility in how you might add other local parts to your system. That you can throw a TCP layer in front of it is awesome -- and that also means you can throw an SCTP (or whatever) layer in front of it if you want later. -Craig From jesper.louis.andersen@REDACTED Sun Nov 29 14:59:53 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 29 Nov 2015 14:59:53 +0100 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: <565AFD3F.9080508@xs4all.nl> References: <565AFD3F.9080508@xs4all.nl> Message-ID: One possible way is to message yourself. The following needs some help because it never loops over the unpacking, which it should. But surely you get the idea. The unpack function is now tasked with returning a list of decoded packets, not just the first one: handle_info({tcp, Socket, RawData}, State_name, #state{buffer_state = BufState} = State) -> inet:setopts(Socket, [{active, once}]), case unpack(BufState, RawData) of {ok, Datas, BufState2} -> [gen_fsm:send_event(self(), {packet, D}) || D <- Datas], {next_state, State_name, State#state { buffer_state = BufState2 }}; {more, BufState2} -> {next_state, State_name, State#state { buffer_state = BufState2 }} end; Now, you can build two variants of the unpack/2 function. Rather than returning {more, State} you can simply return {ok, [], State} which eliminates the need for a separate 'more-variant'. But some times, it is clever to return {more, NeededBytes, State} or something such in order to tell the layer how many bytes it needs before it can produce a new packet. In active mode, this allows you to expect the right number of bytes directly, which sometimes helps since you can move more or the load into the Erlang runtime. Now, the beauty is that the messages will sit in your own mailbox, and in order. So you just handle them as any other messages and move the state accordingly. And when you get new stuff in, your decode it in your handle_info block and inject it into the mailbox. It is a fairly modular solution as well: you effectively decoupled the TCP state from the rest of your code, so you can now freely move it out of the FSM should you prefer doing so. Vice versa, it also decouples the protocol decoding from your FSM state, so it can worry about the semantic impact of the protocol rather than the syntactic impact. On Sun, Nov 29, 2015 at 2:27 PM, Frans Schneider wrote: > Dear list, > > I have a gen_fsm representing the tcp connection to a remote server with 6 > states defined. Tcp data is handled by handle_info with active = once.Tcp > packets can contain more than one commands/data items. > I would like to feed the data into the different FSM states in the same > manner as with send_event() and not handle the different states in separate > handle_info clauses for the different states or a massive handle_info > clause with a huge case statement. > Question is, can I call the state clauses like this or do I brake OTP > behavior?This is a client and blocking incoming data is not an issue. > > handle_info({tcp, Socket, Raw_data}, State_name, #state{buffer = Buffer} = > State) -> > {Datas, Buffer1} = unpack(<>), > lists:foldl(fun(Data, {StateName_in, State_in}) -> > {next_state, StateName_out, State_out} = > ?MODULE:StateName_in({recv, Data}, State_in), > {StateName_out, State_out} > end, {State_name, State}, Datas), > inet:setopts(Socket, [{active, once}]), > {next_state, active, State#state{buffer = Buffer1}}; > > I can of course use a seperate process for handling the tcp stuff which > makes the gen_fsm calls, but was just wondering if this could work as well. > > Thanks, > > /Frans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Sun Nov 29 15:25:37 2015 From: schneider@REDACTED (Frans Schneider) Date: Sun, 29 Nov 2015 15:25:37 +0100 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: References: <565AFD3F.9080508@xs4all.nl> Message-ID: <565B0AE1.4050809@xs4all.nl> Great! thanks for your explanation. Makes sense to send messages instead of calling the state functions. But what about style? Would you consider a solution as you describe good style? I didn't see it in other projects used this way. (Weird thing that good style is becoming such an issue to me since I started coding Erlang.) /Frans On 11/29/2015 02:59 PM, Jesper Louis Andersen wrote: > One possible way is to message yourself. The following needs some help > because it never loops over the unpacking, which it should. But surely > you get the idea. The unpack function is now tasked with returning a > list of decoded packets, not just the first one: > > handle_info({tcp, Socket, RawData}, State_name, #state{buffer_state = > BufState} = State) -> > inet:setopts(Socket, [{active, once}]), > case unpack(BufState, RawData) of > {ok, Datas, BufState2} -> > [gen_fsm:send_event(self(), {packet, D}) || D <- Datas], > {next_state, State_name, State#state { buffer_state = > BufState2 }}; > {more, BufState2} -> > {next_state, State_name, State#state { buffer_state = > BufState2 }} > end; > > Now, you can build two variants of the unpack/2 function. Rather than > returning {more, State} you can simply return {ok, [], State} which > eliminates the need for a separate 'more-variant'. But some times, it > is clever to return {more, NeededBytes, State} or something such in > order to tell the layer how many bytes it needs before it can produce > a new packet. In active mode, this allows you to expect the right > number of bytes directly, which sometimes helps since you can move > more or the load into the Erlang runtime. > > Now, the beauty is that the messages will sit in your own mailbox, and > in order. So you just handle them as any other messages and move the > state accordingly. And when you get new stuff in, your decode it in > your handle_info block and inject it into the mailbox. It is a fairly > modular solution as well: you effectively decoupled the TCP state from > the rest of your code, so you can now freely move it out of the FSM > should you prefer doing so. Vice versa, it also decouples the protocol > decoding from your FSM state, so it can worry about the semantic > impact of the protocol rather than the syntactic impact. > > On Sun, Nov 29, 2015 at 2:27 PM, Frans Schneider > wrote: > > Dear list, > > I have a gen_fsm representing the tcp connection to a remote > server with 6 states defined. Tcp data is handled by handle_info > with active = once.Tcp packets can contain more than one > commands/data items. > I would like to feed the data into the different FSM states in the > same manner as with send_event() and not handle the different > states in separate handle_info clauses for the different states or > a massive handle_info clause with a huge case statement. > Question is, can I call the state clauses like this or do I brake > OTP behavior?This is a client and blocking incoming data is not an > issue. > > handle_info({tcp, Socket, Raw_data}, State_name, #state{buffer = > Buffer} = State) -> > {Datas, Buffer1} = unpack(<>), > lists:foldl(fun(Data, {StateName_in, State_in}) -> > {next_state, StateName_out, State_out} = > ?MODULE:StateName_in({recv, Data}, State_in), > {StateName_out, State_out} > end, {State_name, State}, Datas), > inet:setopts(Socket, [{active, once}]), > {next_state, active, State#state{buffer = Buffer1}}; > > I can of course use a seperate process for handling the tcp stuff > which makes the gen_fsm calls, but was just wondering if this > could work as well. > > Thanks, > > /Frans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -- > J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From freza@REDACTED Sun Nov 29 16:09:12 2015 From: freza@REDACTED (Jachym Holecek) Date: Sun, 29 Nov 2015 10:09:12 -0500 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: <565AFD3F.9080508@xs4all.nl> References: <565AFD3F.9080508@xs4all.nl> Message-ID: <20151129150912.GA29822@circlewave.net> # Frans Schneider 2015-11-29: > I have a gen_fsm representing the tcp connection to a remote server > with 6 states defined. Tcp data is handled by handle_info with > active = once.Tcp packets can contain more than one commands/data > items. > > [...] If your handle_info() were dealing with one PDU at a time, it's fine to do something like: handle_info({tcp, Sock, Data}, State_name, #state{...} = State) -> ..., ?MODULE:State_name(Event, State); Probably not a coincidence that most of the gen_fsm callbacks have the same return convention, allows you to seamlessly transfer control like that. If you find yourself wanting to perform more than one state transition per incoming event, that probably speaks of poorly structured solution. > I can of course use a seperate process for handling the tcp stuff > which makes the gen_fsm calls, but was just wondering if this could > work as well. You're better off spawning a utility process below you FSM, giving it ownership of the socket, then letting it do the reception and framing, sending events to the parent process via send_event(). This utility process should end up really simple and mostly stateless, you probably want plain proc_lib:spawn_link(), keeping the code for it in the same module as the FSM. You'll probably want at least a primitive form of flow control between the child and the parent, for safety, but that's easy (we can get back to that later if you like). Don't know the details of what you're doing, but a generic protocol stacks comment: your receiving process has to handle the full inbound signalling frequency, which makes it a precious shared resource. It should thus do as little work as possible -- with sanely designed protocols this means it would look at constant-size packet header, figure out what to do, and let somebody else decode the payload and perform business logic (or complicated administrative tasks internal to the stack itself). Usually the real work ends up in temporary worker processes (one per request, or one per session if the protocol has such notion, or whatever else makes sense in your context). This maximazes the signalling frequency you can handle, at the same time it helps you spread load across all available cores, also there's the fault isolation angle. HTH, -- Jachym From freza@REDACTED Sun Nov 29 16:27:02 2015 From: freza@REDACTED (Jachym Holecek) Date: Sun, 29 Nov 2015 10:27:02 -0500 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: <565B0AE1.4050809@xs4all.nl> References: <565AFD3F.9080508@xs4all.nl> <565B0AE1.4050809@xs4all.nl> Message-ID: <20151129152702.GB29822@circlewave.net> # Frans Schneider 2015-11-29: > But what about style? Would you consider a solution as you describe > good style? Doesn't look good to me. Firstly it fails to solve the real problem, which was poor functional breakdown of your protocol stack (one process ended up doing too much work). Moreover, it does so at the cost of having to deal with every incoming event twice; introducing message passing overhead without any real gain; risking reordering of incoming events with respect to other internally originated events and finally subjecting actual handling of incoming events to arbitrary latency hit (self-posted message goes at the end of your mailbox -- who knows what's pending there already). And the more load (= higher event frequency) you're handling the more serious will all these will get. BR, -- Jachym From jesper.louis.andersen@REDACTED Sun Nov 29 17:01:27 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 29 Nov 2015 17:01:27 +0100 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: <565B0AE1.4050809@xs4all.nl> References: <565AFD3F.9080508@xs4all.nl> <565B0AE1.4050809@xs4all.nl> Message-ID: On Sun, Nov 29, 2015 at 3:25 PM, Frans Schneider wrote: > But what about style? Would you consider a solution as you describe good > style? I didn't see it in other projects used this way. With the caveats of Jachym in mind: As a stepping stone toward a more layered solution I think it is fine. You usually don't have speed concerns at that point and this somewhat splits the concerns up because you can move the unpacker to its own module. The primary problem is if you can't handle that other messages interleave with the packet processing. But I'm willing to say that this better be handled sooner or later since otherwise you can't move the packet processing out later. Say you always get packets A and B in the order [A, B] on the line. Then, because of TCP, you could get [A] and then [B] later on with a divide in between since the TCP stream got broken up that way. This argues that a message X *can* and *should* be able to interleave, in particular [A, X, B] is possible. Usually processing follows a layered model where the lower layers are more syntax-oriented (i.e., is this even a protobuffed message?) and the higher layers encode semantics (can we receive this packet in that state?). A nice modular program can easily adapt this to different process models, as long as the layers are not intertwined too much. In the case of each session being a single TCP connection, and having perhaps thousands of these, I wouldn't worry too much about separating the decoder out to get more cores working. I'd probably just keep it inside a single process, but calling to a separate module, since you have ample concurrency in the project already. If, on the other hand, you are implementing uTP, where each message arrives on the same socket, then you should heed to words of Jachym and make the decoder loop as fast as possible and forward toward other workers on the inside, essentially setting off a core for decoding and demuxing. Before looking for fast, it is often beneficial to look at correct: processes form nice isolation barriers. So if things start going wrong, what should die and how should it die? This can in certain situations drive you to a solution where it is better if the decoder is spawned in order to handle that part outside of the main semantic state. If the decoder goes wrong, then you are still keeping hold of your semantic state. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sun Nov 29 22:04:27 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sun, 29 Nov 2015 21:04:27 +0000 Subject: [erlang-questions] pattern to augment a release with a plugin system In-Reply-To: <260ACFB9-16CC-4129-BAC0-FC84C22E9F46@gmail.com> References: <260ACFB9-16CC-4129-BAC0-FC84C22E9F46@gmail.com> Message-ID: On Fri, Nov 27, 2015 at 6:04 AM Ulf Wiger wrote: > > On 26 Nov 2015, at 02:37, Benoit Chesneau wrote: > > Are there other ways to do it? What would be the right pattern to do it? > To be honest I am balancing between 2 and 3. 4 would be fine but would mean > rewriting something similar to the release_handler. > > Generally speaking how people feels about augmenting dynamically a release > with new applications/modules ? > > > I?ve been using setup [1] to manage plugins, where each plugin is an > application which you e.g. can copy into a designated plugin directory. > > The code could look like this: > > load_plugins() -> > AllPlugins = find_plugins(), > load_apps(AllPlugins), > register_plugins(AllPlugins), > start_apps(AllPlugins), > {ok, [A || {A,_} <- AllPlugins]}. > > find_plugins() -> > LibDirs = setup:lib_dirs("ERL_SETUP_LIBS"), > AppNames = app_names(LibDirs), > [{A,setup:pick_vsn(A, setup:find_app(A, LibDirs), latest)} || > A <- AppNames]. > > load_apps(AllPlugins) -> > lists:foreach( > fun({A,{V,D}}) -> > true = setup:patch_app(A, V, [D]), > ReloadRes = setup:reload_app(A, V, [D]), > ?debug("reload_app(~p, ~p, ~p) -> ~p~n", [A,V,[D], > ReloadRes]) > end, AllPlugins). > > register_plugins(AllPlugins) -> > ? % provide system-specific code > > start_apps(AllPlugins) -> > lists:foreach( > fun({A, _}) -> > maybe_start(A) > end, AllPlugins). > > maybe_start(App) -> > case application:get_key(App, mod) of > {ok, {_, _}} -> > %% Can be started > application:start(App); > _ -> > ok > end. > > Setup can also help during upgrade, since when you call > setup:reload_app/1, setup can automatically generate upgrade scripts to > transition from a previous version of the app to the newest. > > Another possibly useful feature is that setup can generate new boot > scripts from what?s currently running, with setup:keep_release(SystemVsn). > Not that it was really intented for this scenario, but rather as an easy > way to get started. > > BR, > Ulf W > Thanks! I did to some experiment this week-end using `setup` and so far I am very pleased by what I see. The thing I am currently working on is the "uninstall" action. So far I don't see any code for it in setup (which sounds logical). I do the easy for now : uninstall(myplugin) -> application:stop(myplugin), application:stop(myplugin), remove_path(myplugin). The remove_path function is inspired from the one in setup but not exposed. Although that's the easy way. I probably need to manage possible dependencies. My idea for it is parsing each .app and maintain the relations between each apps. WHich would allows me to stop them as well if they have no more relations. Anyway, I will work more on that topic over the week and come back with some code asap :) Thanks again for your code! - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sun Nov 29 23:49:28 2015 From: mjtruog@REDACTED (Michael Truog) Date: Sun, 29 Nov 2015 14:49:28 -0800 Subject: [erlang-questions] pattern to augment a release with a plugin system In-Reply-To: References: Message-ID: <565B80F8.1000709@gmail.com> On 11/26/2015 02:37 AM, Benoit Chesneau wrote: > Hi all, > > I am actually working on a system to augment a release possibly dynamically with some extensions. > > For now in hooks [1] I am starting applications from the filesystem or just using the applications shipped with the release. I am not a fan of simply starting/stopping applications from the filesystem since for now it is not handling upgrades (only by starting/stopping the application) like a release can do. > > I can see these different ways to handle plugins right now: > > 1) plugins are only added at compile time (something rabbitmq does). Similar to 1 > > pro: plugins are the simple apps > cons: > - the system is not dynamic and a user can't simply augment the application he is using. Only plugins built and shipped with the release are available > > 2) adding plugins to a release and possibly starting them when enabled via the config. The upgrade would be done via a release upgrade. > > Pro: > - doesn't change too much the current way to handle releases upgrade. > - the plugins can be added dynamically > > Cons: > - if i want to add a new plugin I will need to create a new release, increment its version etc. > - Also a new release would have to be build for each users depending on their choice. > - I have to find a way to build a local release based on remote plugins depending on the platform > > 3) load plugins via an external system. If this is an apps, it will be installed using the release_handler, so it could possibly manage the .appup scripts > > pro: plugins are really dynamically handled > > cons: > - not sure on how with the release_handler the installed application will be handled at startup. > - not sure if dynamically augmented release using the release_handler is suppored > > 4) Something similar to 3 , but instead of using the release_handler, i have my own plugin_handler handling upgrades by looking on specific modules functions if needed. > > Are there other ways to do it? What would be the right pattern to do it? To be honest I am balancing between 2 and 3. 4 would be fine but would mean rewriting something similar to the release_handler. > > Generally speaking how people feels about augmenting dynamically a release with new applications/modules ? > > Any feedback is welcome. > > - beno?t > > > [1] https://github.com/barrel-db/hooks > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Having Erlang applications started and stopped dynamically can be contrary to having the system fail-fast. So, that is a danger, since you are not always starting from a known starting state, the dynamic actions are dealing with state that is unknown due to the point in time during the runtime when the actions execute. There is only one boot file which the Erlang VM accepts on its command line, and that contains the list of actions which are suppose to provide a successful startup of the release, and if not, it should crash due to validation during initialization. However, it is helpful to be able to dynamically start and stop Erlang applications, if the system has dynamic functionality it needs to support (hopefully with careful validation in isolation, so it doesn't cause instability). I have functions for this at https://github.com/okeuday/reltool_util . You might like reltool_util:application_remove/3 since that handles dependencies too, if they aren't shared with other applications. I prefer reltool_util:application_start/3 instead of application:ensure_started/1 since it handles module loading too. The reltool_util module also contains functions for handling release boot and script files. Best Regards, Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Nov 30 02:58:25 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 30 Nov 2015 14:58:25 +1300 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: <5657EA88.1010901@gmail.com> References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> <5657EA88.1010901@gmail.com> Message-ID: <1462F19A-19B6-4A16-A26D-4643BC0830CB@cs.otago.ac.nz> On 27/11/2015, at 6:30 pm, Michael Truog wrote: > Having it be opt-in does sound good, but it should complicate > hot-code loading, making a module update possibly fail due to the > import_static dependency on an external module. We already have precisely this problem with headers. Header dependencies are not tracked by the code loading support. If modules A and B rely on header C to describe their interface, and C changes and B is updated to match, the fact that A doesn't match any more is not detected. At least with an import_static dependency *stated* and right there in the .beam file (as it should be) you can *tell* when you have a problem. > If module 'a' uses > import_static on module 'b', does the inlining (actual source code) of > module 'a' change during runtime if module 'b' changes? When the > module 'b' update indirectly alters module 'a', does its module > version not change? If you were allowed to do an update of a > module used for inlining, it could create a ripple-effect in the system, > making the process more brittle and prone to failure, due to the > lack of isolation. Headers are brittle and prone to failure in precisely this way right now, it's just that the system doesn't notice. Hmmm. The lens module could perfectly well be a header. Instead of get({G,_,_}, X) -> G(X). in a module, we'd have -define(GET(Lens, Datum), case Lens of {G,_,_} -> G(Datum) end). and so on. Would this change the actual number or character of dependencies? No. What would be the effect of changing a lens.hrl file instead of a lens.erl file? Just the same breakage except that the system wouldn't notice and would not be able to reject a load telling you something would have broken. It would of course be possible to provide a parse transform for any specific module to accomplish the same effect as cross-module inlining, but this has *precisely* the same dependency effect as cross-module inlining or headers. "We have already determined what you are madam, now we're haggling over the price." I believe that dependence on a parse transform or a header should be tracked in .beam files and that if you try to load a .beam file the system ought to be able to tell you "this has a dependency which has since been changed, why not recompile first?" > > If instead the modules that are inlined are marked "permanent" and > prevented from being updated in the future, then there is a reason > to not use inlining I am not with you. How does making inlining *safe* become a reason to *not* use it? It's not as if I'm proposing anything that wasn't already known in the Lisp world back in the 70s. > > You may be able to make a lens implementation that relies on > header file usage, That's rather like saying "You could avoid the problems of playing Russian Roulette with a pistol by using a fully loaded shotgun instead." > which would allow all the functions to be within a > single module and allowed to be inlined there. That approach > wouldn't cause problems for module updating in the future. It would cause *precisely* the same problems that headers have always (at least potentially) caused. m% cat goo.hrl -define(GOO, 42). m% cat foo.erl -module(foo). -export([foo/0]). -include("goo.hrl"). foo() -> boo:boo(?GOO). m% erlc foo.erl m% ed goo.hrl s/42/137/ w q m% cat goo.hrl -define(GOO, 137). m% cat boo.erl -module(boo). -export([boo/1]). -include("goo.hrl"). boo(?GOO) -> ok. m% erlc erlc boo.erl m% erl 1> l(boo). {module,boo} 2> l(foo). {module,foo} 3> foo:foo(). ** exception error: no function clause matching boo:boo(42) (boo.erl, line 5) 4> c(boo). {ok,boo} 5> c(foo). {ok,foo} 6> foo:foo(). ok RIGHT NOW you can get mysterious errors that go away when you recompile things, due to the use of headers. import_static doesn't increase the problems, or the amount of work you have to do to fix them, or the nature of that work. All it does is give the system a chance to *notice* that the problem already exists. ANY use of information that is not physically present in the source file, whether it comes from using Leex or Yecc or the preprocessor or a parse transform or anything else is a *dependency* that needs to be tracked. From ok@REDACTED Mon Nov 30 03:34:27 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 30 Nov 2015 15:34:27 +1300 Subject: [erlang-questions] Proposed addition to gb_trees In-Reply-To: References: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> Message-ID: <9B4161D1-8664-4DBE-9276-04C584B2A7D4@cs.otago.ac.nz> On 28/11/2015, at 6:55 am, Pierpaolo Bernardi wrote: > On Fri, Nov 27, 2015 at 6:14 AM, Richard A. O'Keefe wrote: > > It should take an optional initializer parameter, like for example the > corresponding function in Racket. > > http://docs.racket-lang.org/reference/hashtables.html#%28def._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._hash-update%29%29 That's a somewhat broken interface. If you want the default result to be a function, you have to wrap an extra function around it, which is nasty. The documentation of hash-update is ambiguous; it is not clear whether the failure-result argument is used as the failure-result of the implicit hash-ref or as the result of hash-update itself. From olopierpa@REDACTED Mon Nov 30 03:47:49 2015 From: olopierpa@REDACTED (Pierpaolo Bernardi) Date: Mon, 30 Nov 2015 03:47:49 +0100 Subject: [erlang-questions] Proposed addition to gb_trees In-Reply-To: <9B4161D1-8664-4DBE-9276-04C584B2A7D4@cs.otago.ac.nz> References: <53E4A33C-E7A6-4704-BCE4-820C771AB792@cs.otago.ac.nz> <9B4161D1-8664-4DBE-9276-04C584B2A7D4@cs.otago.ac.nz> Message-ID: On Mon, Nov 30, 2015 at 3:34 AM, Richard A. O'Keefe wrote: > > On 28/11/2015, at 6:55 am, Pierpaolo Bernardi wrote: > >> On Fri, Nov 27, 2015 at 6:14 AM, Richard A. O'Keefe wrote: >> >> It should take an optional initializer parameter, like for example the >> corresponding function in Racket. >> >> http://docs.racket-lang.org/reference/hashtables.html#%28def._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._hash-update%29%29 > > That's a somewhat broken interface. If you want the > default result to be a function, you have to wrap an > extra function around it, which is nasty. Agreed. Although this case is not common, I think. > The documentation of hash-update is ambiguous; it is > not clear whether the failure-result argument is used > as the failure-result of the implicit hash-ref or as > the result of hash-update itself. Yes, it's not clear from the doc. It's used as the result of the implicit hash-ref. For example: (hash-update table key add1 0) will increment the value associated to key. If key was not present, it will take the value 1 (not 0). From bchesneau@REDACTED Mon Nov 30 05:43:56 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 30 Nov 2015 04:43:56 +0000 Subject: [erlang-questions] pattern to augment a release with a plugin system In-Reply-To: <565B80F8.1000709@gmail.com> References: <565B80F8.1000709@gmail.com> Message-ID: On Sun, Nov 29, 2015 at 11:49 PM Michael Truog wrote: > Having Erlang applications started and stopped dynamically can be contrary > to having the system fail-fast. So, that is a danger, since you are not > always starting from a known starting state, the dynamic actions are > dealing with state that is unknown due to the point in time during the > runtime when the actions execute. There is only one boot file which the > Erlang VM accepts on its command line, and that contains the list of > actions which are suppose to provide a successful startup of the release, > and if not, it should crash due to validation during initialization. > > Right. then maybe "plugins" in Erlang world should be seen as a way to augment a release from binary packages and provide some tooling around to make the experience easy for non Erlang users. Another way could be sending remotely the packages and regenerate locally the boot script. Not sure how crazy it is. I can see the possible uncertainty on reboot if any problem happen during a deployment it can creates though. But how much it would be different from a bad release upgrade? > However, it is helpful to be able to dynamically start and stop Erlang > applications, if the system has dynamic functionality it needs to support > (hopefully with careful validation in isolation, so it doesn't cause > instability). I have functions for this at > https://github.com/okeuday/reltool_util . You might like > reltool_util:application_remove/3 since that handles dependencies too, if > they aren't shared with other applications. I prefer > reltool_util:application_start/3 instead of application:ensure_started/1 > since it handles module loading too. The reltool_util module also contains > functions for handling release boot and script files. > Thanks this is useful :) With the help of setup, i have some working code right now to enable/disable dynamically the applications using remote plugins. - benoit > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Mon Nov 30 06:50:37 2015 From: zxq9@REDACTED (zxq9) Date: Mon, 30 Nov 2015 14:50:37 +0900 Subject: [erlang-questions] pattern to augment a release with a plugin system In-Reply-To: References: <565B80F8.1000709@gmail.com> Message-ID: <4602134.X1CJFEa0gC@changa> On 2015?11?30? ??? 04:43:56 Benoit Chesneau wrote: > On Sun, Nov 29, 2015 at 11:49 PM Michael Truog wrote: > > > Having Erlang applications started and stopped dynamically can be contrary > > to having the system fail-fast. So, that is a danger, since you are not > > always starting from a known starting state, the dynamic actions are > > dealing with state that is unknown due to the point in time during the > > runtime when the actions execute. There is only one boot file which the > > Erlang VM accepts on its command line, and that contains the list of > > actions which are suppose to provide a successful startup of the release, > > and if not, it should crash due to validation during initialization. > > > > > Right. then maybe "plugins" in Erlang world should be seen as a way to > augment a release from binary packages and provide some tooling around to > make the experience easy for non Erlang users. > > Another way could be sending remotely the packages and regenerate locally > the boot script. Not sure how crazy it is. I can see the possible > uncertainty on reboot if any problem happen during a deployment it can > creates though. But how much it would be different from a bad release > upgrade? One way it differes: you expect a reboot, you don't expect a relup failure. The circus that necessarily surrounds trying to make plugins work with releases is one reason why (for client-side code where we need plugins) we release signed source bundles, have clients checkupdate/build on execution, and execute from the file system the way other language runtimes tend to. We're not deploying to telecom switches or industrial controllers, and we aren't expecting live upgrades to work. Note, that is for *client* side code -- a pretty uncommon use case across Erlangdom. I have a hard time seeing the advantage of a server-side dynamic plugin system based on releases that isn't distributed as a configurable bundle of source ([config plugins] && [run release building tool]). The hardest part of relups is making sure it won't break anything. It seems like it would require either some very strict coding conventions (and version control) or some very exhaustive live integration testing to determine whether or not a relup from R1{P1, P2} to R2{P1, P2, P3} would work, not to mention the case of R1{P1, P2} to R2{P2, P3}. The amount of work involved in that would probably squash any benefits of even having a plugin system, not to mention make a rolling upgrade look easy in comparison to a relup. You would either need a fancy version numbering system or a local way of tracking version numbers, too... hrm. Releases don't seem very plugin-friendly. What am I missing? -Craig From max.lapshin@REDACTED Mon Nov 30 08:12:37 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 30 Nov 2015 14:12:37 +0700 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: References: <565AFD3F.9080508@xs4all.nl> <565B0AE1.4050809@xs4all.nl> Message-ID: We have fully refused from using gen_fsm and especially in handling protocol states in gen_fsm. Everywhere we use something like: handle_info({tcp, Socket, Input}, #state{protocol_state = S0} = State) -> {ok, OutputCommands, S1} = some_protocol:handle(Input, S0), ..handle somehow OutputCommands inet:setopts(Socket, [{active,once}]), {noreply, State#state{protocol_state = S1}}. Such approach allows to record Input to file and write a test for output commands. Also it allows more easily handle situation when there is more than 1 command in Input packet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Mon Nov 30 08:41:36 2015 From: schneider@REDACTED (Frans Schneider) Date: Mon, 30 Nov 2015 08:41:36 +0100 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: References: <565AFD3F.9080508@xs4all.nl> <565B0AE1.4050809@xs4all.nl> Message-ID: <565BFDB0.5080007@xs4all.nl> For me, implementing the protocol as a gen_fsm feels very natural. Since the unpacking of my data involves splitting the raw data stream in proper chunks, conditionally decrypting these chunks with a stream cypher and lastly decoding them into stanzas according to some mutilated protocol scheme, I think I will use a stack of processes, each specific for the task at hand. Having modules / processes dealing with one particular task feels rather good. Thanks for all the input, which is a great help to me. /Frans On 11/30/2015 08:12 AM, Max Lapshin wrote: > We have fully refused from using gen_fsm and especially in handling > protocol states in gen_fsm. > > Everywhere we use something like: > > handle_info({tcp, Socket, Input}, #state{protocol_state = S0} = State) -> > {ok, OutputCommands, S1} = some_protocol:handle(Input, S0), > ..handle somehow OutputCommands > inet:setopts(Socket, [{active,once}]), > {noreply, State#state{protocol_state = S1}}. > > > Such approach allows to record Input to file and write a test for > output commands. Also it allows more easily handle situation when > there is more than 1 command in Input packet. > > From bchesneau@REDACTED Mon Nov 30 08:42:53 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 30 Nov 2015 07:42:53 +0000 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: <565BFDB0.5080007@xs4all.nl> References: <565AFD3F.9080508@xs4all.nl> <565B0AE1.4050809@xs4all.nl> <565BFDB0.5080007@xs4all.nl> Message-ID: isn't the ssl module using a gen_fsm? On Mon, 30 Nov 2015 at 08:41, Frans Schneider wrote: > For me, implementing the protocol as a gen_fsm feels very natural. Since > the unpacking of my data involves splitting the raw data stream in > proper chunks, conditionally decrypting these chunks with a stream > cypher and lastly decoding them into stanzas according to some mutilated > protocol scheme, I think I will use a stack of processes, each specific > for the task at hand. Having modules / processes dealing with one > particular task feels rather good. > > Thanks for all the input, which is a great help to me. > > /Frans > > On 11/30/2015 08:12 AM, Max Lapshin wrote: > > We have fully refused from using gen_fsm and especially in handling > > protocol states in gen_fsm. > > > > Everywhere we use something like: > > > > handle_info({tcp, Socket, Input}, #state{protocol_state = S0} = State) > -> > > {ok, OutputCommands, S1} = some_protocol:handle(Input, S0), > > ..handle somehow OutputCommands > > inet:setopts(Socket, [{active,once}]), > > {noreply, State#state{protocol_state = S1}}. > > > > > > Such approach allows to record Input to file and write a test for > > output commands. Also it allows more easily handle situation when > > there is more than 1 command in Input packet. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean.parpaillon@REDACTED Mon Nov 30 10:19:46 2015 From: jean.parpaillon@REDACTED (Jean Parpaillon) Date: Mon, 30 Nov 2015 10:19:46 +0100 Subject: [erlang-questions] erlang D-Bus implementation ? In-Reply-To: <56589354.60304@gmail.com> References: <1448641716.4193.81.camel@free.fr> <56589354.60304@gmail.com> Message-ID: <1448875186.10386.39.camel@free.fr> Hi, Le vendredi 27 novembre 2015 ? 09:31 -0800, Michael Truog a ?crit?: > On 11/27/2015 08:28 AM, Jean Parpaillon wrote: > > Hi all, > > There have been several tentative to have a pure erlang > > implementation > > of D-Bus RPC system, most of them forks of (unmaintained) http://so > > urce > > forge.net/projects/dbus-erlang/ > > > > I have also forked it some years ago and use it in another project. > > I > > can't say it's perfectly operational but at least for consuming D- > > Bus > > services it works. Exposing D-Bus services is not possible. The > > original erlang-dbus was proposing a gen_dbus behaviour for that > > but I > > preferred to focus on robust service consuming first. > > > > My version is there: https://github.com/lizenn/erlang-dbus/ > > > > TODO list: > > - documentation ;) > > - testing: test infra is operational (with travis-ci) but coverage > > is > > really low > > - gen_dbus coding > > - well know interfaces coding: Properties (partly coded in > > dbus_properties_proxy), ObjectManager, ... > > > > In a far future, we could imagine OTP to include it, as DBus is > > used > > for many services and as cros-language RPC system. The only problem > > regarding OTP is most DBus implementations runs on top of UNIX > > sockets, > > which requires platform dependant code. OTP version could be > > operational with TCP sockets (implemented in erlang-dbus), with an > > option of communicating through UNIX socket with ports. > > > > Is anybody interested in contributing ? > You can track UNIX socket support at https://github.com/erlang/otp/pu > ll/612 . > Thanks for the link. BTW, I'm currently using procket + inert for UDS [0] and would be glad to migrate to anything more standard (OTP) as soon as available, even if procket + inert works currently fine. Cheers, [0]?https://github.com/lizenn/erlang-dbus/blob/master/src/dbus_transpor t_unix.erl -- Jean Parpaillon -- Open Source Consultant OW2 Technology Council Chairman Director @ OW2 Consortium OCCIware Strategic Orientation Committee Chairman Research Engineer @ Inria -- Phone: +33 6 30 10 92 86 im: jean.parpaillon@REDACTED skype: jean.parpaillon linkedin: http://www.linkedin.com/in/jeanparpaillon/en From ingela.andin@REDACTED Mon Nov 30 10:42:14 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 30 Nov 2015 10:42:14 +0100 Subject: [erlang-questions] tcp data in a gen_fsm In-Reply-To: References: <565AFD3F.9080508@xs4all.nl> <565B0AE1.4050809@xs4all.nl> <565BFDB0.5080007@xs4all.nl> Message-ID: Hi! 2015-11-30 8:42 GMT+01:00 Benoit Chesneau : > isn't the ssl module using a gen_fsm? > > Sort of! The protocol of course fits the FSM-principal and the states helps understanding the code. But the state functions are called by a local dispatch function as all events so to speak are recived on the socket. We thought of sending a message to ourselfs, but that does not really work out as data recived on the socket may need to be treated diffrently due to data received just before, and then sending a message to ourselfs may cause the data that need to be handled in the new way to be recived before the event that changes the state telling us how to treat the new data. We are actually looking into making a new fsm behaviour as the current one make too many assumptions of the world beening pure Erlang that just do not fit the most common use case. Regards Ingela Erlang/OTP team - Ericsson AB > On Mon, 30 Nov 2015 at 08:41, Frans Schneider wrote: > >> For me, implementing the protocol as a gen_fsm feels very natural. Since >> the unpacking of my data involves splitting the raw data stream in >> proper chunks, conditionally decrypting these chunks with a stream >> cypher and lastly decoding them into stanzas according to some mutilated >> protocol scheme, I think I will use a stack of processes, each specific >> for the task at hand. Having modules / processes dealing with one >> particular task feels rather good. >> >> Thanks for all the input, which is a great help to me. >> >> /Frans >> >> On 11/30/2015 08:12 AM, Max Lapshin wrote: >> > We have fully refused from using gen_fsm and especially in handling >> > protocol states in gen_fsm. >> > >> > Everywhere we use something like: >> > >> > handle_info({tcp, Socket, Input}, #state{protocol_state = S0} = State) >> -> >> > {ok, OutputCommands, S1} = some_protocol:handle(Input, S0), >> > ..handle somehow OutputCommands >> > inet:setopts(Socket, [{active,once}]), >> > {noreply, State#state{protocol_state = S1}}. >> > >> > >> > Such approach allows to record Input to file and write a test for >> > output commands. Also it allows more easily handle situation when >> > there is more than 1 command in Input packet. >> > >> > >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Mon Nov 30 20:42:36 2015 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 30 Nov 2015 11:42:36 -0800 Subject: [erlang-questions] Updates, lenses, and why cross-module inlining would be nice In-Reply-To: <1462F19A-19B6-4A16-A26D-4643BC0830CB@cs.otago.ac.nz> References: <1448544949.1214768.450737385.156BF7D2@webmail.messagingengine.com> <5657EA88.1010901@gmail.com> <1462F19A-19B6-4A16-A26D-4643BC0830CB@cs.otago.ac.nz> Message-ID: <565CA6AC.9020306@gmail.com> On 11/29/2015 05:58 PM, Richard A. O'Keefe wrote: > On 27/11/2015, at 6:30 pm, Michael Truog wrote: >> Having it be opt-in does sound good, but it should complicate >> hot-code loading, making a module update possibly fail due to the >> import_static dependency on an external module. > We already have precisely this problem with headers. > Header dependencies are not tracked by the code loading > support. If modules A and B rely on header C to > describe their interface, and C changes and B is updated > to match, the fact that A doesn't match any more is not > detected. > > At least with an import_static dependency *stated* and > right there in the .beam file (as it should be) you can > *tell* when you have a problem. The header approach is preferable to make the dependency problems a compile-time concern, rather than creating a runtime failure when attempting a module update. It is important to not crash running source code due to errors. > >> If module 'a' uses >> import_static on module 'b', does the inlining (actual source code) of >> module 'a' change during runtime if module 'b' changes? When the >> module 'b' update indirectly alters module 'a', does its module >> version not change? If you were allowed to do an update of a >> module used for inlining, it could create a ripple-effect in the system, >> making the process more brittle and prone to failure, due to the >> lack of isolation. > Headers are brittle and prone to failure in precisely this > way right now, it's just that the system doesn't notice. > > Hmmm. The lens module could perfectly well be a header. > Instead of > > get({G,_,_}, X) -> G(X). > > in a module, we'd have > > -define(GET(Lens, Datum), case Lens of {G,_,_} -> G(Datum) end). > > and so on. Would this change the actual number or character of > dependencies? No. What would be the effect of changing a > lens.hrl file instead of a lens.erl file? Just the same breakage > except that the system wouldn't notice and would not be able to > reject a load telling you something would have broken. I was thinking about a slightly different approach, which I adapted your module to at https://github.com/okeuday/lens/ . The header file is at https://github.com/okeuday/lens/blob/master/include/lens.hrl . The main changes can be summarized as: 1) change all functions to have a "lens_" prefix 2) add a compile nowarn_unused_function directive with all the interface functions (comment: unfortunately the F/A syntax is not supported here right now, but it is supported in the inline compile directive) 3) move the example to an eunit test at https://github.com/okeuday/lens/blob/master/test/lens_test.erl I put the module example and the header example in the same eunit test to show that the main difference visually is using a "_" instead of a ":". The inlining creates warnings due to terms not being used from the header file, which is annoying, but shows that the inlining works (I don't believe there is a way to turn off that warning, currently). I was using rebar2 to build/test. The point of this, is to show the same source code can be used, with inlining, and that the potential for breakage can be handled with testing all the functions. A lens interface should not change a whole lot, so it can be a dependable interface that is trusted. > > It would of course be possible to provide a parse transform > for any specific module to accomplish the same effect as > cross-module inlining, but this has *precisely* the same > dependency effect as cross-module inlining or headers. > > "We have already determined what you are madam, now we're > haggling over the price." > > I believe that dependence on a parse transform or a header > should be tracked in .beam files and that if you try to > load a .beam file the system ought to be able to tell you > "this has a dependency which has since been changed, why > not recompile first?" This would be interesting, but would likely need to use a hash of the parse transform contents and header file contents. I don't think it would be very helpful though, since parse transforms and header files are only used during compilation and the build should be checking timestamps to see if files change. If it tracked the preprocessor input, it could say "yes, this preprocessor input changed" or "no, this preprocessor input did not change", but you can't determine what the changes were from the .beam addition or anything beyond that. What do you do with the information? >> If instead the modules that are inlined are marked "permanent" and >> prevented from being updated in the future, then there is a reason >> to not use inlining > I am not with you. How does making inlining *safe* become a > reason to *not* use it? > > It's not as if I'm proposing anything that wasn't already known > in the Lisp world back in the 70s. If modules are marked permanent due to module inlining, it will be preventing modules from changing in the future. If it is done as you have described, it would be controlled by higher-level source code, preventing lower-level source code from changing, making development less flexible. It works fine, until it doesn't, then you have people renaming modules to avoid the problem, and trying to maintain more than 1 copy of the source code. I see that as a way to cause problems, not solve them. >> You may be able to make a lens implementation that relies on >> header file usage, > That's rather like saying "You could avoid the problems of > playing Russian Roulette with a pistol by using a fully > loaded shotgun instead." > >> which would allow all the functions to be within a >> single module and allowed to be inlined there. That approach >> wouldn't cause problems for module updating in the future. > It would cause *precisely* the same problems that headers > have always (at least potentially) caused. > > m% cat goo.hrl > -define(GOO, 42). > > m% cat foo.erl > -module(foo). > -export([foo/0]). > -include("goo.hrl"). > > foo() -> > boo:boo(?GOO). > > m% erlc foo.erl > m% ed goo.hrl > s/42/137/ > w > q > m% cat goo.hrl > -define(GOO, 137). > > m% cat boo.erl > -module(boo). > -export([boo/1]). > -include("goo.hrl"). > > boo(?GOO) -> > ok. > m% erlc erlc boo.erl > m% erl > > 1> l(boo). > {module,boo} > 2> l(foo). > {module,foo} > 3> foo:foo(). > ** exception error: no function clause matching boo:boo(42) (boo.erl, line 5) > 4> c(boo). > {ok,boo} > 5> c(foo). > {ok,foo} > 6> foo:foo(). > ok > > RIGHT NOW you can get mysterious errors that go away > when you recompile things, due to the use of headers. > import_static doesn't increase the problems, or the > amount of work you have to do to fix them, or the nature > of that work. All it does is give the system a chance > to *notice* that the problem already exists. Yes. I believe this doesn't become a concern when tests are provided. > > ANY use of information that is not physically present in > the source file, whether it comes from using Leex or Yecc > or the preprocessor or a parse transform or anything else > is a *dependency* that needs to be tracked. Ok, but what runtime problem does this solve? From lloyd@REDACTED Mon Nov 30 21:09:00 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Mon, 30 Nov 2015 15:09:00 -0500 (EST) Subject: [erlang-questions] erlpass issue Message-ID: <1448914140.023416224@apps.rackspace.com> Hi Fred, Hope all is well and life is giving you many grins. Sorry to bother you, but I've run into the following mystery with erlpass: I've configured erlpass as a dependency in different two Nitrogen apps--- wgb and wgc. Here, respectively, are the two rebar configurations: wgb--- {erlpass, "", {git, "git://github.com/ferd/erlpass", {branch, master}}}, wgc--- {erlpass, "", {git, "git://github.com/ferd/erlpass", {branch, master}}}, To my eye they look to be the same. But, when I execute make, wgb compiles: make[1]: Leaving directory `/home/lloyd/wgb/lib/bcrypt/c_src' ==> erlpass (compile) ==> nitrogen_tinymce (compile) But wgc doesn't: ==> erlpass (compile) ERROR: compile failed while processing /home/lloyd/wgc/lib/erlpass: {'EXIT',{badarg,[{io,format, [<0.24.0>,"Dependency not available: ~p-~s (~p)\n", [bcrypt, {git,"https://github.com/opscode/erlang-bcrypt.git", {branch,"master"}}, undefined]], []}, {lists,foreach,2,[{file,"lists.erl"},{line,1337}]}, {rebar_deps,do_check_deps,1,[]}, {rebar_deps,compile,2,[]}, {rebar_core,run_modules,4,[]}, {rebar_core,execute,6,[]}, {rebar_core,maybe_execute,8,[]}, {rebar_core,process_dir1,7,[]}]}} make: *** [compile] Error 1 And I'm stuck. Any thoughts on how clear up this strange state of affairs? All the best, Lloyd From t@REDACTED Mon Nov 30 21:24:59 2015 From: t@REDACTED (Tristan Sloughter) Date: Mon, 30 Nov 2015 14:24:59 -0600 Subject: [erlang-questions] erlpass issue In-Reply-To: <1448914140.023416224@apps.rackspace.com> References: <1448914140.023416224@apps.rackspace.com> Message-ID: <1448915099.2505849.453918745.3D34A5D8@webmail.messagingengine.com> erlpass' rebar config uses the rebar3 format of deps which leaves off the second element in the tuple: https://github.com/ferd/erlpass/blob/master/rebar.config#L1-L2 This causes the crash you see. -- Tristan Sloughter t@REDACTED On Mon, Nov 30, 2015, at 02:09 PM, lloyd@REDACTED wrote: > Hi Fred, > > Hope all is well and life is giving you many grins. Sorry to bother you, > but I've run into the following mystery with erlpass: > > I've configured erlpass as a dependency in different two Nitrogen apps--- > wgb and wgc. > > Here, respectively, are the two rebar configurations: > > wgb--- > > {erlpass, "", {git, "git://github.com/ferd/erlpass", > {branch, master}}}, > > wgc--- > > {erlpass, "", {git, "git://github.com/ferd/erlpass", > {branch, master}}}, > > To my eye they look to be the same. But, when I execute make, wgb > compiles: > > make[1]: Leaving directory `/home/lloyd/wgb/lib/bcrypt/c_src' > ==> erlpass (compile) > ==> nitrogen_tinymce (compile) > > But wgc doesn't: > > ==> erlpass (compile) > ERROR: compile failed while processing /home/lloyd/wgc/lib/erlpass: > {'EXIT',{badarg,[{io,format, > [<0.24.0>,"Dependency not available: ~p-~s (~p)\n", > [bcrypt, > {git,"https://github.com/opscode/erlang-bcrypt.git", > {branch,"master"}}, > undefined]], > []}, > {lists,foreach,2,[{file,"lists.erl"},{line,1337}]}, > {rebar_deps,do_check_deps,1,[]}, > {rebar_deps,compile,2,[]}, > {rebar_core,run_modules,4,[]}, > {rebar_core,execute,6,[]}, > {rebar_core,maybe_execute,8,[]}, > {rebar_core,process_dir1,7,[]}]}} > make: *** [compile] Error 1 > > And I'm stuck. Any thoughts on how clear up this strange state of > affairs? > > All the best, > > Lloyd > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Mon Nov 30 21:28:55 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 30 Nov 2015 15:28:55 -0500 Subject: [erlang-questions] erlpass issue In-Reply-To: <1448914140.023416224@apps.rackspace.com> References: <1448914140.023416224@apps.rackspace.com> Message-ID: <20151130202853.GB886@fhebert-ltm1> If you're still on rebar2, use the prior-to-last tagged version. I recently bumped it to rebar3 and that broke rebar2 support inadvertently. If using rebar2, you should *never* rely on the master branch because the tool itself does not support automatically converting that to a stable reference. Rebar3 would instead generate a lock file that ensures you keep using the proper version, though. Let me know if that helps. Regard, Fred. On 11/30, lloyd@REDACTED wrote: >Hi Fred, > >Hope all is well and life is giving you many grins. Sorry to bother you, but I've run into the following mystery with erlpass: > >I've configured erlpass as a dependency in different two Nitrogen apps--- wgb and wgc. > >Here, respectively, are the two rebar configurations: > >wgb--- > > {erlpass, "", {git, "git://github.com/ferd/erlpass", {branch, master}}}, > >wgc--- > > {erlpass, "", {git, "git://github.com/ferd/erlpass", {branch, master}}}, > >To my eye they look to be the same. But, when I execute make, wgb compiles: > >make[1]: Leaving directory `/home/lloyd/wgb/lib/bcrypt/c_src' >==> erlpass (compile) >==> nitrogen_tinymce (compile) > >But wgc doesn't: > >==> erlpass (compile) >ERROR: compile failed while processing /home/lloyd/wgc/lib/erlpass: {'EXIT',{badarg,[{io,format, > [<0.24.0>,"Dependency not available: ~p-~s (~p)\n", > [bcrypt, > {git,"https://github.com/opscode/erlang-bcrypt.git", > {branch,"master"}}, > undefined]], > []}, > {lists,foreach,2,[{file,"lists.erl"},{line,1337}]}, > {rebar_deps,do_check_deps,1,[]}, > {rebar_deps,compile,2,[]}, > {rebar_core,run_modules,4,[]}, > {rebar_core,execute,6,[]}, > {rebar_core,maybe_execute,8,[]}, > {rebar_core,process_dir1,7,[]}]}} >make: *** [compile] Error 1 > >And I'm stuck. Any thoughts on how clear up this strange state of affairs? > >All the best, > >Lloyd > > > From dleach@REDACTED Mon Nov 30 22:25:55 2015 From: dleach@REDACTED (David Leach) Date: Mon, 30 Nov 2015 21:25:55 +0000 Subject: [erlang-questions] Supervision question Message-ID: Hi All, We have an application that starts pools of different kinds of workers and we want to offer the caller the ability to "link" to a pool of workers. We're using poolboy to manage the pools and gproc for naming the pools and ensuring startup is complete before returning a reference to the caller. We don't want to link to poolboy, because it can and should be restarted by the supervisor supervising it. The behavior we want is, if the supervisor managing poolboy reaches its maximum restarts the caller who is linked also dies. We also want that pool to disappear if the caller dies and the caller can restart it again. Structure looks a little like this: pool_manager super_sup -> worker_sup -> poolboy Right now, the caller is linked to the worker_sup so that if the worker_sup goes down caller also goes down. The pool_manager monitors the caller and sends exit(WorkerSupPid, shutdown) if the caller goes down so that the pool gets cleaned up. Is there a better way of doing this? Better being easier for someone reading the code to understand quickly or more reliably if there is a problem with this implementation? We thought about having the super_sup start two children, one of them performing the role of the pool_manager in this situation . But that seems a bit excessive, and how might one name it clearly to differentiate the roles of the pool_manager and it. The pool manager takes care of details of the pool and translates the reference the caller has to a pid. Thanks in Advance, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From seancribbs@REDACTED Mon Nov 30 22:58:57 2015 From: seancribbs@REDACTED (Sean Cribbs) Date: Mon, 30 Nov 2015 15:58:57 -0600 Subject: [erlang-questions] Supervision question In-Reply-To: References: Message-ID: If I'm reading this correctly, your worker_sup will be restarted after the max_restarts is reached for the poolboy pool. This means that the caller can link or monitor worker_sup, correct? Or do you need a one_for_all strategy? I have a suspicion dependencies (link/monitor) and discovery (gproc, registered names) are also being confused or conflated in your question. On Mon, Nov 30, 2015 at 3:25 PM, David Leach wrote: > Hi All, > > > We have an application that starts pools of different kinds of workers and > we want to offer the caller the ability to "link" to a pool of workers. > We're using poolboy to manage the pools and gproc for naming the pools and > ensuring startup is complete before returning a reference to the caller. We > don't want to link to poolboy, because it can and should be restarted by > the supervisor supervising it. The behavior we want is, if the supervisor > managing poolboy reaches its maximum restarts the caller who is linked > also dies. We also want that pool to disappear if the caller dies and the > caller can restart it again. Structure looks a little like this: > > > pool_manager > > super_sup -> worker_sup -> > > poolboy > > > Right now, the caller is linked to the worker_sup so that if the > worker_sup goes down caller also goes down. The pool_manager monitors the > caller and sends exit(WorkerSupPid, shutdown) if the caller goes down so > that the pool gets cleaned up. > > > Is there a better way of doing this? Better being easier for someone > reading the code to understand quickly or more reliably if there is a > problem with this implementation? > > > We thought about having the super_sup start two children, one of > them performing the role of the pool_manager in this situation . But that > seems a bit excessive, and how might one name it clearly to differentiate > the roles of the pool_manager and it. The pool manager takes care of > details of the pool and translates the reference the caller has to a pid. > > > Thanks in Advance, > > > David > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dleach@REDACTED Mon Nov 30 23:12:40 2015 From: dleach@REDACTED (David Leach) Date: Mon, 30 Nov 2015 22:12:40 +0000 Subject: [erlang-questions] Supervision question In-Reply-To: References: , Message-ID: Yes you're right, now that I re-read my question it's not particularly relevant. worker_sup will not be restarted by super_sup if the max restarts for poolboy pool is reached. A caller will call: start_pool_link(Opts) This triggers the super_sup to start a worker_sup which starts poolboy. The caller is linked to worker_sup, so if it goes down (because it dies or because the max restart is reached) the caller will also go down. If the caller goes down, the pool_managager which is monitoring the caller will take worker_sup down with exit(WorkerSupPid, shutdown). Resorting to the monitor is because worker_sup won't go down when the caller exits. This is so that in the situation where either the caller or the pool goes down all processes are cleaned up. Let me know if you need any further clarification. ________________________________ From: Sean Cribbs Sent: Tuesday, 1 December 2015 10:58 a.m. To: David Leach Cc: Erlang Questions Subject: Re: [erlang-questions] Supervision question If I'm reading this correctly, your worker_sup will be restarted after the max_restarts is reached for the poolboy pool. This means that the caller can link or monitor worker_sup, correct? Or do you need a one_for_all strategy? I have a suspicion dependencies (link/monitor) and discovery (gproc, registered names) are also being confused or conflated in your question. On Mon, Nov 30, 2015 at 3:25 PM, David Leach > wrote: Hi All, We have an application that starts pools of different kinds of workers and we want to offer the caller the ability to "link" to a pool of workers. We're using poolboy to manage the pools and gproc for naming the pools and ensuring startup is complete before returning a reference to the caller. We don't want to link to poolboy, because it can and should be restarted by the supervisor supervising it. The behavior we want is, if the supervisor managing poolboy reaches its maximum restarts the caller who is linked also dies. We also want that pool to disappear if the caller dies and the caller can restart it again. Structure looks a little like this: pool_manager super_sup -> worker_sup -> poolboy Right now, the caller is linked to the worker_sup so that if the worker_sup goes down caller also goes down. The pool_manager monitors the caller and sends exit(WorkerSupPid, shutdown) if the caller goes down so that the pool gets cleaned up. Is there a better way of doing this? Better being easier for someone reading the code to understand quickly or more reliably if there is a problem with this implementation? We thought about having the super_sup start two children, one of them performing the role of the pool_manager in this situation . But that seems a bit excessive, and how might one name it clearly to differentiate the roles of the pool_manager and it. The pool manager takes care of details of the pool and translates the reference the caller has to a pid. Thanks in Advance, David _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: