From n.oxyde@REDACTED Tue Aug 1 13:58:08 2017 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 1 Aug 2017 13:58:08 +0200 Subject: [erlang-questions] Internal compiler atoms In-Reply-To: References: Message-ID: <1F3DD675-42AB-4DF4-8868-ED8221320637@gmail.com> > Le 27 juil. 2017 ? 16:19, H?kan Mattsson a ?crit : > > > I am trying to compile a ?(?big?)? module from a list of forms??, but it seems? like the compiler internally generates lots of new atoms:? > > no more index entries in atom_tab (max=1048576) > > Crash dump is being written to: erl_crash.dump... > > ?In this case the compiler itself generated ?more than 300K atoms while compiling my forms. > ? > ?Why is the atoms generated?? > > ?Is this anything that can be disabled?? > > /H?kan Write a core_transform that replaces all atoms in c_vars by integers. Given the Core inliner already emits variables named after integer values, the rest of the Core compiler passes should be able to cope with them. > > Le 28 juil. 2017 ? 08:43, Albin Stig? a ?crit : > > 4. If you really DO need to create dynamic atoms for a quick and dirty > hack, keep in mind that atoms with a common prefix ie. foo_1, foo_2, > foo_3 etc will lead worse performance because of how erlang compares > atoms (some Erlang guru correct me if I'm wrong but this used to be > the case). Wrong, atom comparison is always O(1). From n.oxyde@REDACTED Tue Aug 1 14:01:51 2017 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 1 Aug 2017 14:01:51 +0200 Subject: [erlang-questions] Internal compiler atoms In-Reply-To: <1F3DD675-42AB-4DF4-8868-ED8221320637@gmail.com> References: <1F3DD675-42AB-4DF4-8868-ED8221320637@gmail.com> Message-ID: <9BB51E88-9DF8-4682-95C5-1269AACE8910@gmail.com> > Le 1 ao?t 2017 ? 13:58, Anthony Ramine a ?crit : > > Write a core_transform that replaces all atoms in c_vars by integers. Given the Core inliner already emits variables named after integer values, the rest of the Core compiler passes should be able to cope with them. Actually I just realised that wouldn't solve the problem, because the problem is probably that one of the Core passes explicitly emits atomic names for nothing. The good news is that given the inliner emits integers, we can do the same in the culprit pass. From mikpelinux@REDACTED Tue Aug 1 15:11:13 2017 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Tue, 1 Aug 2017 15:11:13 +0200 Subject: [erlang-questions] Internal compiler atoms In-Reply-To: <1F3DD675-42AB-4DF4-8868-ED8221320637@gmail.com> References: <1F3DD675-42AB-4DF4-8868-ED8221320637@gmail.com> Message-ID: <22912.32241.405030.809703@gargle.gargle.HOWL> Anthony Ramine writes: > > Le 28 juil. 2017 ? 08:43, Albin Stig? a ?crit : > > > > 4. If you really DO need to create dynamic atoms for a quick and dirty > > hack, keep in mind that atoms with a common prefix ie. foo_1, foo_2, > > foo_3 etc will lead worse performance because of how erlang compares > > atoms (some Erlang guru correct me if I'm wrong but this used to be > > the case). > > Wrong, atom comparison is always O(1). Equality/inequality tests on atoms is O(1). (That's the whole point of atoms, and their analogues in LISP, i.e. "symbols".) Comparisons (<, >, etc) is O(length(atom_to_list(Atom)), and with common prefixes the cost is indeed higher. C.f. erts/emulator/beam/utils.c:erts_cmp_atoms(). From pepm.workshop@REDACTED Wed Aug 2 01:52:43 2017 From: pepm.workshop@REDACTED (PEPM Workshop) Date: Wed, 2 Aug 2017 08:52:43 +0900 Subject: [erlang-questions] PEPM 2018 Call for Papers Message-ID: -- CALL FOR PAPERS -- ACM SIGPLAN Workshop on PARTIAL EVALUATION AND PROGRAM MANIPULATION (PEPM) 2018 =============================================================================== * Website : http://popl18.sigplan.org/track/PEPM-2018 * Time : 8th ? 9th January 2018 * Place : Los Angeles, CA, US (co-located with POPL 2018) The ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation (PEPM), which has a history going back to 1991 and has co-located with POPL every year since 2006, originates in the discoveries of practically useful automated techniques for evaluating programs with only partial input. Over the years, the scope of PEPM has expanded to include a variety of research areas centred around the theme of semantics-based program manipulation ? the systematic exploitation of treating programs not only as subject to black-box execution, but also as data structures that can be generated, analysed, and transformed while establishing or maintaining important semantic properties. Scope ----- In addition to the traditional PEPM topics (see below), PEPM 2018 welcomes submissions in new domains, in particular: * Semantics based and machine-learning based program synthesis and program optimisation. * Modelling, analysis, and transformation techniques for distributed and concurrent protocols and programs, such as session types, linear types, and contract specifications. More generally, topics of interest for PEPM 2018 include, but are not limited to: * Program and model manipulation techniques such as: supercompilation, partial evaluation, fusion, on-the-fly program adaptation, active libraries, program inversion, slicing, symbolic execution, refactoring, decompilation, and obfuscation. * Techniques that treat programs/models as data objects including metaprogramming, generative programming, embedded domain-specific languages, program synthesis by sketching and inductive programming, staged computation, and model-driven program generation and transformation. * Program analysis techniques that are used to drive program/model manipulation such as: abstract interpretation, termination checking, binding-time analysis, constraint solving, type systems, automated testing and test case generation. * Application of the above techniques including case studies of program manipulation in real-world (industrial, open-source) projects and software development processes, descriptions of robust tools capable of effectively handling realistic applications, benchmarking. Examples of application domains include legacy program understanding and transformation, DSL implementations, visual languages and end-user programming, scientific computing, middleware frameworks and infrastructure needed for distributed and web-based applications, embedded and resource-limited computation, and security. This list of categories is not exhaustive, and we encourage submissions describing new theories and applications related to semantics-based program manipulation in general. If you have a question as to whether a potential submission is within the scope of the workshop, please contact the programme co-chairs, Fritz Henglein (http://www.diku.dk/~henglein/) and Josh Ko (https://josh-hs-ko.github.io). Submission categories and guidelines ------------------------------------ Two kinds of submissions will be accepted: Regular Research Papers and Short Papers. * Regular Research Papers should describe new results, and will be judged on originality, correctness, significance, and clarity. Regular research papers must not exceed 12 pages (excluding bibliography). * Short Papers may include tool demonstrations and presentations of exciting if not fully polished research, and of interesting academic, industrial, and open-source applications that are new or unfamiliar. Short papers must not exceed 6 pages (excluding bibliography). Both kinds of submissions should be typeset using the two-column ?sigplan? sub-format of the new ?acmart? format available at: http://sigplan.org/Resources/Author/ and submitted electronically via HotCRP: https://pepm18.hotcrp.com/ PEPM 2018 will employ lightweight double-blind reviewing according to the rules of POPL 2018. Quoting from POPL 2018?s call for papers: ?submitted papers must adhere to two rules: 1. author names and institutions must be omitted, and 2. references to authors? own related work should be in the third person (e.g., not ?We build on our previous work ...? but rather ?We build on the work of ...?). The purpose of this process is to help the PC and external reviewers come to an initial judgment about the paper without bias, not to make it impossible for them to discover the authors if they were to try. Nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult. In particular, important background references should not be omitted or anonymized. In addition, authors should feel free to disseminate their ideas or draft versions of their paper as they normally would. For instance, authors may post drafts of their papers on the web or give talks on their research ideas.? See POPL 2018?s Submission and Reviewing FAQ page for more information: http://popl18.sigplan.org/track/POPL-2018-papers#Submission-and-Reviewing-FAQ Submissions are welcome from PC members (except the two co-chairs) provided that there are non-PC co-authors. Accepted papers will appear in formal proceedings published by ACM, and be included in the ACM Digital Library. Authors of short papers, however, can ask for their papers to be left out of the formal proceedings. At least one author of each accepted contribution must attend the workshop and present the work. In the case of tool demonstration papers, a live demonstration of the described tool is expected. Suggested topics, evaluation criteria, and writing guidelines for both research tool demonstration papers will be made available on the PEPM 2018 web site. Student participants with accepted papers can apply for a SIGPLAN PAC grant to help cover travel expenses and other support. PAC also offers other support, such as for child-care expenses during the meeting or for travel costs for companions of SIGPLAN members with physical disabilities, as well as for travel from locations outside of North America and Europe. For details on the PAC programme, see its web page. Important dates --------------- * Paper submission deadline : Friday 6th October 2017 (AoE) (firm) * Author notification : Saturday 4th November 2017 * Workshop : Monday 8th ? Tuesday 9th January 2018 The proceedings will be published 2 weeks pre-conference. AUTHORS TAKE NOTE: The official publication date is the date the proceedings are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of your conference. The official publication date affects the deadline for any patent filings related to published work. (For those rare conferences whose proceedings are published in the ACM Digital Library after the conference is over, the official publication date remains the first day of the conference.) Best paper award ---------------- PEPM 2018 continues the tradition of a Best Paper award. The winner will be announced at the workshop. Programme committee ------------------- Nada Amin (EPFL) Shigeru Chiba (University of Tokyo) Ezgi ?i?ek (Max Planck Institute for Software Systems) Olivier Danvy (Yale-NUS College) Ronald Garcia (University of British Columbia) Simon Gay (University of Glasgow) Andy Gill (X, the Moonshot Factory) Fritz Henglein (co-chair) (University of Copenhagen) Anastasia Izmaylova (IMC Financial Markets) Johan Jeuring (Utrecht University) Gabriele Keller (University of New South Wales) Oleg Kiselyov (Tohoku University) Hsiang-Shang Ko (co-chair) (National Institute of Informatics) Ralf L?mmel (University of Koblenz-Landau) Julia Lawall (Inria) Simon Peyton Jones (Microsoft Research Cambridge) Frank Pfenning (Carnegie Mellon University) Sriram Rajamani (Microsoft Research India) Norman Ramsey (Tufts University) Thomas Reps (University of Wisconsin-Madison) Sergei Romanenko (Keldysh Institute of Applied Mathematics) Tiark Rompf (Purdue University) Wolfram Schulte (Facebook) Peter Sestoft (IT University of Copenhagen) Harald S?ndergaard (University of Melbourne) Kohei Suenaga (Kyoto University) Martin Vechev (ETH Zurich) Marcos Viera (University of the Republic) Nobuko Yoshida (Imperial College London) From getonga@REDACTED Wed Aug 2 13:43:33 2017 From: getonga@REDACTED (=?utf-8?B?Z2V0b25nYQ==?=) Date: Wed, 2 Aug 2017 19:43:33 +0800 Subject: [erlang-questions] mochiweb in erlang 20 problem Message-ID: I compile mochiweb in erlang 20, and I find this error: ``` src/mochiweb_html.erl:8: Warning: export_all flag enabled - all functions will be exported Compiled src/mochiweb_html.erl src/mochiweb_multipart.erl:41: Warning: crypto:rand_bytes/1 is deprecated and will be removed in a future release; use crypto:strong_rand_bytes/1 Compiled src/mochiweb_multipart.erl ``` I made a pr to the mochiweb, but not accepted for a long time and not been discussed. Which erlang version do you use with mochiweb? Does the mochiweb still in maintenance? -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz@REDACTED Wed Aug 2 18:35:19 2017 From: aschultz@REDACTED (Andreas Schultz) Date: Wed, 2 Aug 2017 18:35:19 +0200 (CEST) Subject: [erlang-questions] binary match vs. is_binary in optimization Message-ID: <246001762.826034.1501691719574.JavaMail.zimbra@tpip.net> Hi, Can someone explain why the binary optimization treats a is_binary() guard different that an binary match? I the attached test code, I get a: > test.erl:7: Warning: NOT OPTIMIZED: called function t1/1 does not begin with a suitable binary matching instruction and a > test.erl:14: Warning: OPTIMIZED: creation of sub binary delayed for the second function. IMHO both should be identical, but apparently they are not. Many thanks, Andreas Test code: -module(test). -compile([bin_opt_info]). -export([do/1]). do(<>) -> t1(Bin), t3(Bin). t1(Bin) when is_binary(Bin) -> test(Bin, []). t3(<>) -> test(Bin, []). test(<>, _) -> X. -- Dipl.-Inform. Andreas Schultz email: as@REDACTED phone: +49-391-819099-224 ----------------------- enabling your networks ---------------------- Travelping GmbH phone: +49-391-81 90 99 0 Roentgenstr. 13 fax: +49-391-81 90 99 299 39108 Magdeburg email: info@REDACTED GERMANY web: http://www.travelping.com Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 --------------------------------------------------------------------- From alex0player@REDACTED Wed Aug 2 22:45:12 2017 From: alex0player@REDACTED (Alex S.) Date: Wed, 2 Aug 2017 23:45:12 +0300 Subject: [erlang-questions] binary match vs. is_binary in optimization In-Reply-To: <246001762.826034.1501691719574.JavaMail.zimbra@tpip.net> References: <246001762.826034.1501691719574.JavaMail.zimbra@tpip.net> Message-ID: t1 doesn't extract a sub-binary from your binary, so there's nothing to optimize. 2017-08-02 19:35 GMT+03:00 Andreas Schultz : > Hi, > > Can someone explain why the binary optimization treats a is_binary() > guard different that an binary match? > > I the attached test code, I get a: > > > test.erl:7: Warning: NOT OPTIMIZED: called function t1/1 does not begin > with a suitable binary matching instruction > > and a > > > test.erl:14: Warning: OPTIMIZED: creation of sub binary delayed > > for the second function. IMHO both should be identical, but apparently > they are not. > > Many thanks, > Andreas > > Test code: > > -module(test). > > -compile([bin_opt_info]). > > -export([do/1]). > > do(<>) -> > t1(Bin), > t3(Bin). > > t1(Bin) when is_binary(Bin) -> > test(Bin, []). > > t3(<>) -> > test(Bin, []). > > test(<>, _) -> > X. > -- > Dipl.-Inform. Andreas Schultz > > email: as@REDACTED > phone: +49-391-819099-224 > > ----------------------- enabling your networks ---------------------- > > Travelping GmbH phone: +49-391-81 90 99 0 > Roentgenstr. 13 fax: +49-391-81 90 99 299 > 39108 Magdeburg email: info@REDACTED > GERMANY web: http://www.travelping.com > > Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 > Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 > --------------------------------------------------------------------- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juanjo@REDACTED Thu Aug 3 00:41:42 2017 From: juanjo@REDACTED (Juan Jose Comellas) Date: Wed, 2 Aug 2017 19:41:42 -0300 Subject: [erlang-questions] mochiweb in erlang 20 problem In-Reply-To: References: Message-ID: Those are innocuous warnings at the moment. Maybe mochiweb is not being updated to support much older versions of Erlang. Still, if this bothers you too much or you don't want to maintain your own fork of mochiweb I'd suggest switching to a more modern HTTP server library such as Cowboy . You'll also gain support for websockets and eventually HTTP 2. On Wed, Aug 2, 2017 at 8:43 AM, getonga wrote: > I compile mochiweb in erlang 20, and I find this error: > ``` > src/mochiweb_html.erl:8: Warning: export_all flag enabled - all functions > will be exported > Compiled src/mochiweb_html.erl > src/mochiweb_multipart.erl:41: Warning: crypto:rand_bytes/1 is deprecated > and will be removed in a future release; use crypto:strong_rand_bytes/1 > Compiled src/mochiweb_multipart.erl > ``` > > I made a pr to the mochiweb, but not accepted for a long time and not been > discussed. > Which erlang version do you use with mochiweb? > Does the mochiweb still in maintenance? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From krzysztof.jurewicz@REDACTED Thu Aug 3 14:59:39 2017 From: krzysztof.jurewicz@REDACTED (Krzysztof Jurewicz) Date: Thu, 03 Aug 2017 14:59:39 +0200 Subject: [erlang-questions] Writing a cryptocurrency in Erlang In-Reply-To: <87lgn820gn.fsf@gmail.com> References: <87lgn820gn.fsf@gmail.com> Message-ID: <87y3r0kds4.fsf@gmail.com> > I?ve been writing a new cryptocurrency in Erlang. [?] I?ve published the cleaned code at https://gitlab.com/Ercoin/ercoin . Contributions and feedback are welcome. From max.lapshin@REDACTED Fri Aug 4 08:06:26 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 4 Aug 2017 09:06:26 +0300 Subject: [erlang-questions] erlang 20.0 under elbrus: compilation problems Message-ID: Hi. trying to build erlang 20 under elbrus computer (arch e2k), it goes with compiler lcc trying to configure with: ./configure --prefix=/opt/flussonic --enable-smp-support --enable-kernel-poll --enable-hipe --without-java --enable-dirty-schedulers erts/configure fails, in erts/config.log I see: | #define SIZEOF_TIME_T 8 | /* end confdefs.h. */ | | /* pr36339.c */ | extern void abort (void); | | typedef unsigned long my_uintptr_t; | | int check_a(my_uintptr_t tagged_ptr); | | int __attribute__((noinline)) try_a(my_uintptr_t x) | { | my_uintptr_t heap[2]; | my_uintptr_t *hp = heap; | | hp[0] = x; | hp[1] = 0; | return check_a((my_uintptr_t)(void*)((char*)hp + 1)); | } | | int __attribute__((noinline)) check_a(my_uintptr_t tagged_ptr) | { | my_uintptr_t *hp = (my_uintptr_t*)(void*)((char*)tagged_ptr - 1); | | if (hp[0] == 42 && hp[1] == 0) | return 0; | return -1; | } | | int main(void) | { | if (try_a(42) < 0) | abort (); | return 0; | } | configure:34880: result: yes configure:34883: error: This gcc miscompiles the Erlang runtime system; please use a different version I have commented this place in configure and after some other patches it has compiled. Is it a fatal error? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Fri Aug 4 09:45:22 2017 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Fri, 4 Aug 2017 09:45:22 +0200 Subject: [erlang-questions] erlang 20.0 under elbrus: compilation problems In-Reply-To: References: Message-ID: <22916.9746.939984.249344@gargle.gargle.HOWL> Max Lapshin writes: > Hi. > > trying to build erlang 20 under elbrus computer (arch e2k), it goes with > compiler lcc > > trying to configure with: > > ./configure --prefix=/opt/flussonic --enable-smp-support > --enable-kernel-poll --enable-hipe --without-java --enable-dirty-schedulers > > erts/configure fails, in erts/config.log I see: > > > > > | #define SIZEOF_TIME_T 8 > > | /* end confdefs.h. */ > > | > > | /* pr36339.c */ > > | extern void abort (void); > > | > > | typedef unsigned long my_uintptr_t; > > | > > | int check_a(my_uintptr_t tagged_ptr); > > | > > | int __attribute__((noinline)) try_a(my_uintptr_t x) > > | { > > | my_uintptr_t heap[2]; > > | my_uintptr_t *hp = heap; > > | > > | hp[0] = x; > > | hp[1] = 0; > > | return check_a((my_uintptr_t)(void*)((char*)hp + 1)); > > | } > > | > > | int __attribute__((noinline)) check_a(my_uintptr_t tagged_ptr) > > | { > > | my_uintptr_t *hp = (my_uintptr_t*)(void*)((char*)tagged_ptr - 1); > > | > > | if (hp[0] == 42 && hp[1] == 0) > > | return 0; > > | return -1; > > | } > > | > > | int main(void) > > | { > > | if (try_a(42) < 0) > > | abort (); > > | return 0; > > | } > > | > > configure:34880: result: yes > > configure:34883: error: This gcc miscompiles the Erlang runtime system; > please use a different version > > > I have commented this place in configure and after some other patches it > has compiled. > > > Is it a fatal error? Yes. This is a test for GCC's bug PR36339, which appeared with gcc-4.3.0 and which broke the Erlang VM. Note that it actually checks for the miscompile, not for a compiler version, so the bug is really present. The solution, as configure correctly says, is to use a different compiler. From silviu.cpp@REDACTED Sat Aug 5 09:25:20 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Sat, 5 Aug 2017 10:25:20 +0300 Subject: [erlang-questions] Writing a cryptocurrency in Erlang In-Reply-To: <87y3r0kds4.fsf@gmail.com> References: <87lgn820gn.fsf@gmail.com> <87y3r0kds4.fsf@gmail.com> Message-ID: Why not github ? Easy to track for most of the peoples .. Silviu On Thu, Aug 3, 2017 at 3:59 PM, Krzysztof Jurewicz < krzysztof.jurewicz@REDACTED> wrote: > > I?ve been writing a new cryptocurrency in Erlang. [?] > > I?ve published the cleaned code at https://gitlab.com/Ercoin/ercoin . > Contributions and feedback are welcome. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hc@REDACTED Sat Aug 5 15:59:57 2017 From: hc@REDACTED (H.C. v. Stockhausen) Date: Sat, 05 Aug 2017 13:59:57 +0000 Subject: [erlang-questions] Writing a cryptocurrency in Erlang In-Reply-To: References: <87lgn820gn.fsf@gmail.com> <87y3r0kds4.fsf@gmail.com> Message-ID: I agree. Signed up for gitlab now but ... On Sat, 05 Aug 2017, 09:25 Caragea Silviu, wrote: > Why not github ? Easy to track for most of the peoples .. > > Silviu > > On Thu, Aug 3, 2017 at 3:59 PM, Krzysztof Jurewicz < > krzysztof.jurewicz@REDACTED> wrote: > >> > I?ve been writing a new cryptocurrency in Erlang. [?] >> >> I?ve published the cleaned code at https://gitlab.com/Ercoin/ercoin . >> Contributions and feedback are welcome. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sat Aug 5 16:56:53 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Sat, 5 Aug 2017 16:56:53 +0200 Subject: [erlang-questions] Writing a cryptocurrency in Erlang In-Reply-To: References: <87lgn820gn.fsf@gmail.com> <87y3r0kds4.fsf@gmail.com> Message-ID: You can always set multiple targets for your git remotes and when you push make it land on github, gitlab and others. I have stopped using Github as my primary platform a while ago and nobody really noticed. On 08/05/2017 03:59 PM, H.C. v. Stockhausen wrote: > I agree. Signed up for gitlab now but ... > > On Sat, 05 Aug 2017, 09:25 Caragea Silviu, > wrote: > > Why not github ? Easy to track for most of the peoples .. > > Silviu > > On Thu, Aug 3, 2017 at 3:59 PM, Krzysztof Jurewicz > > > wrote: > > > I?ve been writing a new cryptocurrency in Erlang. [?] > > I?ve published the cleaned code at > https://gitlab.com/Ercoin/ercoin . Contributions and feedback > are welcome. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From libnl@REDACTED Sun Aug 6 13:44:58 2017 From: libnl@REDACTED (lib nl) Date: Sun, 6 Aug 2017 11:44:58 +0000 Subject: [erlang-questions] escript exception error, undefined function, why can't use fun in escript script? Message-ID: Hi, On Linux Mint 18.2 + Erlang/OTP 20.0, there are two escript scripts: a and b. 1. a #!/usr/bin/env escript main(_) -> L = lists:map(fun(N) -> 2*N end, lists:seq(1,10)), io:format("~p~n", [L]). 2. b #!/usr/bin/env escript main(_) -> L = lists:map(fun f2/1, lists:seq(1,10)), io:format("~p~n", [L]). f2(N) -> 2 * N. $ chmod u+x a $ ./a [2,4,6,8,10,12,14,16,18,20] $ chmod u+x b $ ./b escript: exception error: undefined function erl_eval:f2/1 So why can't use fun in escript script? Is this a limitation or known issue? Are there any references that can give more info regarding this behavior? Thanks! Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From karolis.velicka@REDACTED Sun Aug 6 15:44:13 2017 From: karolis.velicka@REDACTED (Karl Velicka) Date: Sun, 6 Aug 2017 14:44:13 +0100 Subject: [erlang-questions] escript exception error, undefined function, why can't use fun in escript script? In-Reply-To: References: Message-ID: Hi, I've run into this myself once - it seems that you can't have functions other than main/1 in escripts, although this doesn't appear to be documented. The way around this is to make your scripts compiled, these can have multiple functions defined in them. You'll need to add this line somewhere in your .escript file: -mode(compile). Cheers, Karl On 6 August 2017 at 12:44, lib nl wrote: > Hi, > > > > On Linux Mint 18.2 + Erlang/OTP 20.0, there are two escript scripts: a and > b. > > > > 1. a > > #!/usr/bin/env escript > > main(_) -> > > L = lists:map(fun(N) -> 2*N end, lists:seq(1,10)), > > io:format("~p~n", [L]). > > > > > > 2. b > > #!/usr/bin/env escript > > main(_) -> > > L = lists:map(fun f2/1, lists:seq(1,10)), > > io:format("~p~n", [L]). > > > > f2(N) -> 2 * N. > > > > > > $ chmod u+x a > > $ ./a > > [2,4,6,8,10,12,14,16,18,20] > > > > $ chmod u+x b > > $ ./b > > escript: exception error: undefined function erl_eval:f2/1 > > > > > > So why can?t use fun in escript script? Is this a limitation or known issue? > Are there any references that can give more info regarding this behavior? > Thanks! > > > > > > > > Thanks! > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From hawk.mattsson@REDACTED Sun Aug 6 16:22:33 2017 From: hawk.mattsson@REDACTED (=?UTF-8?Q?H=C3=A5kan_Mattsson?=) Date: Sun, 6 Aug 2017 16:22:33 +0200 Subject: [erlang-questions] escript exception error, undefined function, why can't use fun in escript script? In-Reply-To: References: Message-ID: Yes, you may have other functions than main/1 in escripts. But the escript must be compiled. ?? Either compiled in run-time ?(using -mode(compile) ?)? ?or pre-compiled ?(using a beam file or archive). /H?kan On Sun, Aug 6, 2017 at 3:44 PM, Karl Velicka wrote: > Hi, > > I've run into this myself once - it seems that you can't have > functions other than main/1 in escripts, although this doesn't appear > to be documented. > > The way around this is to make your scripts compiled, these can have > multiple functions defined in them. You'll need to add this line > somewhere in your .escript file: > > -mode(compile). > > > Cheers, > Karl > > On 6 August 2017 at 12:44, lib nl wrote: > > Hi, > > > > > > > > On Linux Mint 18.2 + Erlang/OTP 20.0, there are two escript scripts: a > and > > b. > > > > > > > > 1. a > > > > #!/usr/bin/env escript > > > > main(_) -> > > > > L = lists:map(fun(N) -> 2*N end, lists:seq(1,10)), > > > > io:format("~p~n", [L]). > > > > > > > > > > > > 2. b > > > > #!/usr/bin/env escript > > > > main(_) -> > > > > L = lists:map(fun f2/1, lists:seq(1,10)), > > > > io:format("~p~n", [L]). > > > > > > > > f2(N) -> 2 * N. > > > > > > > > > > > > $ chmod u+x a > > > > $ ./a > > > > [2,4,6,8,10,12,14,16,18,20] > > > > > > > > $ chmod u+x b > > > > $ ./b > > > > escript: exception error: undefined function erl_eval:f2/1 > > > > > > > > > > > > So why can?t use fun in escript script? Is this a limitation or known > issue? > > Are there any references that can give more info regarding this behavior? > > Thanks! > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang.org@REDACTED Sun Aug 6 16:45:20 2017 From: erlang.org@REDACTED (Stanislaw Klekot) Date: Sun, 6 Aug 2017 16:45:20 +0200 Subject: [erlang-questions] escript exception error, undefined function, why can't use fun in escript script? In-Reply-To: References: Message-ID: <20170806144520.GA6790@jarowit.net> On Sun, Aug 06, 2017 at 04:22:33PM +0200, H?kan Mattsson wrote: > Yes, you may have other functions than main/1 in escripts. But the escript > must be compiled. > ?? > Either compiled in run-time > ?(using -mode(compile) > ?)? > ?or pre-compiled ?(using a beam file or archive). Actually no, you don't need the escript to be compiled to have functions in there. They work just fine, it's just that they cannot be referenced with `fun foo/X' syntax if the escript script is not marked as compiled. -- Stanislaw Klekot From krzysztof.jurewicz@REDACTED Sun Aug 6 20:58:17 2017 From: krzysztof.jurewicz@REDACTED (Krzysztof Jurewicz) Date: Sun, 06 Aug 2017 20:58:17 +0200 Subject: [erlang-questions] Writing a cryptocurrency in Erlang In-Reply-To: References: <87lgn820gn.fsf@gmail.com> <87y3r0kds4.fsf@gmail.com> Message-ID: <874ltkjzg6.fsf@gmail.com> Having a GitHub mirror can be done, but it would ease tracking only of the repository itself, not of related issues. Tracking full project is possible without registration using an Atom feed ( https://gitlab.com/Ercoin/ercoin.atom ); there are also Atom feeds for branches ( https://gitlab.com/Ercoin/ercoin/commits/master?format=atom ) and raw repository can potentially be tracked using plain Git. It is also possible to login to GitLab using GitHub. I?m not saying however that there will no GitHub mirror. It can be argued that it is easier to gain audience when hosting a repository on a more popular service, but I don?t believe that a solution to this problem is making everyone just use the one, biggest site (GitHub in this case). If people were able to create decentralized social networking sites and protocols like ActivityPub, then it should be also possible to create a decentralized social coding site. Actually there already exists a P2P GitHub alternative using Secure Scuttlebutt: https://git.scuttlebot.io/%25RPKzL382v2fAia5HuDNHD5kkFdlP7bGvXQApSXqOBwc%3D.sha256 , but currently it seems to suffer from important deficiencies: https://viewer.scuttlebot.io/%25aN0VDFB0KsqhaouPoVzQzm6HBcKAvlExRH6pR%2F9Khww=.sha256 From silviu.cpp@REDACTED Sun Aug 6 22:18:29 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Sun, 6 Aug 2017 23:18:29 +0300 Subject: [erlang-questions] Writing a cryptocurrency in Erlang In-Reply-To: <874ltkjzg6.fsf@gmail.com> References: <87lgn820gn.fsf@gmail.com> <87y3r0kds4.fsf@gmail.com> <874ltkjzg6.fsf@gmail.com> Message-ID: Hello, I personally hate to create accounts over accounts for all kind of services that are doing the same stuff, I usually use only github and star or watch the projects that are really nice to receive notifications when updates are done. I think at least a github mirror will be nice for your project Silviu On Sun, Aug 6, 2017 at 9:58 PM, Krzysztof Jurewicz < krzysztof.jurewicz@REDACTED> wrote: > Having a GitHub mirror can be done, but it would ease tracking only of the > repository itself, not of related issues. Tracking full project is possible > without registration using an Atom feed ( https://gitlab.com/Ercoin/ > ercoin.atom ); there are also Atom feeds for branches ( > https://gitlab.com/Ercoin/ercoin/commits/master?format=atom ) and raw > repository can potentially be tracked using plain Git. It is also possible > to login to GitLab using GitHub. I?m not saying however that there will no > GitHub mirror. > > It can be argued that it is easier to gain audience when hosting a > repository on a more popular service, but I don?t believe that a solution > to this problem is making everyone just use the one, biggest site (GitHub > in this case). If people were able to create decentralized social > networking sites and protocols like ActivityPub, then it should be also > possible to create a decentralized social coding site. Actually there > already exists a P2P GitHub alternative using Secure Scuttlebutt: > https://git.scuttlebot.io/%25RPKzL382v2fAia5HuDNHD5kkFdlP > 7bGvXQApSXqOBwc%3D.sha256 , but currently it seems to suffer from > important deficiencies: https://viewer.scuttlebot.io/% > 25aN0VDFB0KsqhaouPoVzQzm6HBcKAvlExRH6pR%2F9Khww=.sha256 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From liuzhongzheng2012@REDACTED Mon Aug 7 03:41:24 2017 From: liuzhongzheng2012@REDACTED (Zhongzheng Liu) Date: Mon, 7 Aug 2017 09:41:24 +0800 Subject: [erlang-questions] Should erlang have macro FUNCTION_ARGS ? Message-ID: Hi mail-list: We have FUNCTION_NAME and FUNCTION_ARITY in OTP 19+. Is there FUNCTION_ARGS to get the argument list in current functions? For example: a_function(X, Y) -> ?FUNCTION_ARGS. After preprocessing, the example will look like: a_function(X, Y) -> [X, Y]. Using this macro, we can use erlang:apply(fun ?FUNCTION_NAME/?FUNCTION_ARITY, ?FUNCTION_ARGS) to make any function recursively. For example: -define(RECURSIVE_FUNCION, erlang:apply(fun ?FUNCTION_NAME/?FUNCTION_ARITY, ?FUNCTION_ARGS)). b_function(A) -> do_something(), ?RECURSIVE_FUNCION. After preprocessing, the example will look like: b_function(A) -> do_something(), erlang:apply(fun b_function/1, [A]). Using this macro, we can use io:format("~p", [FUNCTION_ARGS]) to print arguments in any function. For example: -define(PRINT_ARGS, io:format("~p", [FUNCTION_ARGS])). c_function(A) -> PRINT_ARGS, do_something(A). After preprocessing, the example will look like: b_function(A) -> io:format("~p", [A]), erlang:apply(fun b_function/1, [A]). From ok@REDACTED Mon Aug 7 04:39:03 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 7 Aug 2017 14:39:03 +1200 Subject: [erlang-questions] Should erlang have macro FUNCTION_ARGS ? In-Reply-To: References: Message-ID: <9FFB84B3-E708-412D-8598-64B625F0C200@cs.otago.ac.nz> > On 7/08/2017, at 1:41 PM, Zhongzheng Liu wrote: > > Hi mail-list: > > We have FUNCTION_NAME and FUNCTION_ARITY in OTP 19+. > > Is there FUNCTION_ARGS to get the argument list in current functions? What is "THE argument list"? Suppose I have foo(X, [a,b,c], 42) -> ... foo(0, Y, 127) -> ... which clause defines *THE* argument list? > > Using this macro, we can use erlang:apply(fun > ?FUNCTION_NAME/?FUNCTION_ARITY, ?FUNCTION_ARGS) to make any function > recursively. Unfortunately, the only kind of recursion you can create this way is the kind you don't want. Since any variables there might happen to be in the arguments cannot have changed, the new call has *exactly* the same arguments as the old, and will do *exactly* the same as the original call, including the recursive call. > After preprocessing, the example will look like: > > b_function(A) -> > do_something(), > erlang:apply(fun b_function/1, [A]). This is a non-terminating function, for example. > > Using this macro, we can use io:format("~p", [FUNCTION_ARGS]) to print > arguments in any function. True. Considering that arguments can be extremely large, just how often do we want to print all the arguments unselectively? I don't think this *CAN* work in the presence of maps. In a pattern (like the argument list), you have to write #{ key1 := val1, ..., keyn := valn } but that syntax is not allowed as an expression. You have to write #{ key1 => val1, ..., keyn => valn } for that. For example: d_function(#{ a => X }) -> % syntax error here e_function(#{a => X}); % ok d_function(#{ b := Y }) -> % ok e_function(#{b := Y}). % syntax error here I am not thrilled about this discrepancy. No other language I know does this, and it means that I can define a macro -define(MAGIC_VALUE, ...any ground term...). that stands for a ground term, and I can use that macro freely in patterns AND expressions, EXCEPT IF IT CONTAINS A MAP. But that is the way it is, like it or not. From bjorn@REDACTED Mon Aug 7 07:27:13 2017 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 7 Aug 2017 07:27:13 +0200 Subject: [erlang-questions] core erlang apply requires variable in function position? In-Reply-To: <22888.49341.541459.110669@gargle.gargle.HOWL> References: <22888.49341.541459.110669@gargle.gargle.HOWL> Message-ID: On Fri, Jul 14, 2017 at 3:01 PM, Mikael Pettersson wrote: [...] > So is this restriction deliberate or unintentional? Unintentional. sys_core_fold is supposed to handle any legal Core Erlang code, even if it has not be produced by v3_core. Here is a fix: https://github.com/erlang/otp/pull/1526 /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From libnl@REDACTED Mon Aug 7 00:42:13 2017 From: libnl@REDACTED (nl lib) Date: Sun, 6 Aug 2017 22:42:13 +0000 Subject: [erlang-questions] escript exception error, undefined function, why can't use fun in escript script? Message-ID: After adding the following line -mode(compile). It works like a charm. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Mon Aug 7 10:10:35 2017 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 7 Aug 2017 10:10:35 +0200 Subject: [erlang-questions] Internal compiler atoms In-Reply-To: References: Message-ID: On Thu, Jul 27, 2017 at 4:19 PM, H?kan Mattsson wrote: > > I am trying to compile a > ( > big > ) > module from a list of forms > , but it seems > like the compiler internally generates lots of new atoms: > > > no more index entries in atom_tab (max=1048576) > > Crash dump is being written to: erl_crash.dump... Please show us the code (a scaled-down representative sample is fine). /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From liuzhongzheng2012@REDACTED Mon Aug 7 10:22:53 2017 From: liuzhongzheng2012@REDACTED (Zhongzheng Liu) Date: Mon, 7 Aug 2017 16:22:53 +0800 Subject: [erlang-questions] Should erlang have macro FUNCTION_ARGS ? Message-ID: Hi Richard: > What is "THE argument list"? > Suppose I have > foo(X, [a,b,c], 42) -> ... > foo(0, Y, 127) -> ... > which clause defines *THE* argument list? The argument list will change its value in different clause, just like ?LINE. > > > > Using this macro, we can use io:format("~p", [FUNCTION_ARGS]) to print > > arguments in any function. > True. Considering that arguments can be extremely large, > just how often do we want to print all the arguments unselectively? Using io:format/2 to print arguments is just a example to explain the usage. In practice I can send arguments to tx(https://github.com/kvakvs/tx) or store them in somewhere for further analysis. > I don't think this *CAN* work in the presence of maps. > In a pattern (like the argument list), you have to write > #{ key1 := val1, ..., keyn := valn } > but that syntax is not allowed as an expression. You have > to write > #{ key1 => val1, ..., keyn => valn } > for that. For example: > d_function(#{ a => X }) -> % syntax error here > e_function(#{a => X}); % ok > d_function(#{ b := Y }) -> % ok > e_function(#{b := Y}). % syntax error here > I am not thrilled about this discrepancy. No other language > I know does this, and it means that I can define a macro > -define(MAGIC_VALUE, ...any ground term...). > that stands for a ground term, and I can use that macro > freely in patterns AND expressions, > EXCEPT IF IT CONTAINS A MAP. > But that is the way it is, like it or not. I think it can work with temporary variable. foo(X, [a,b,c], 42) -> ?FUNCTION_ARGS. will become: foo(X=__TEMP_1__, [a,b,c]=__TEMP_2__, #{ b := Y }=__TEMP_3__) -> [__TEMP_1__, __TEMP_2__, __TEMP_3__] . It will work with maps. From max.lapshin@REDACTED Mon Aug 7 11:12:12 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 7 Aug 2017 12:12:12 +0300 Subject: [erlang-questions] erlang 20.0 under elbrus: compilation problems In-Reply-To: <22916.9746.939984.249344@gargle.gargle.HOWL> References: <22916.9746.939984.249344@gargle.gargle.HOWL> Message-ID: Got it, thanks. It is working with this buggy compiler, so I will inform compiler authors about this problem. It is not gcc, but lcc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew@REDACTED Mon Aug 7 14:13:23 2017 From: andrew@REDACTED (Andrew McIntyre) Date: Mon, 7 Aug 2017 22:13:23 +1000 Subject: [erlang-questions] List Question Message-ID: <1288613114.20170807221323@Medical-Objects.com.au> Hello All, A Newbie question, Can I tell the difference Between a List of Strings and a Single String? eg the Value in a List may just be "FT" or it might be ["FT", "ST"] I do not know in advance if its a single string or a list of strings, but want to behave differently - actually return the first element of the list when there are multiple values. I have parsed a variable hierarchical structure and the hierarchy can vary over instances. As strings are lists "is_list" returns true for both. Is that soluble or do I need to change the data structure to indicate it. Sorry if this has obvious answer, but new to erlang. Thanks -- Best regards, Andrew McIntyre mailto:andrew@REDACTED sent from a real computer R&D Director Medical-Objects 4/102 Wises Road MAROOCHYDORE Q 4558 AUSTRALIA www.medical-objects.com.au From zxq9@REDACTED Mon Aug 7 14:24:37 2017 From: zxq9@REDACTED (zxq9) Date: Mon, 07 Aug 2017 21:24:37 +0900 Subject: [erlang-questions] List Question In-Reply-To: <1288613114.20170807221323@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> Message-ID: <35768330.vdyX1H0hul@changa> On 2017?08?07? ??? 22:13:23 Andrew McIntyre wrote: > Hello All, > > A Newbie question, > > Can I tell the difference Between a List of Strings and a Single > String? > > eg the Value in a List may just be "FT" or it might be ["FT", "ST"] Using a match this is pretty straight forward. choose_your_adventure("FT") -> path_one(); choose_your_adventure(["FT" | Rest]) -> ok = path_two(), choose_your_adventure(Rest). Generally speaking, if your system is the one generating the lists/strings then it is useful to produce them in a way that is more semantically unambiguous. -Craig From dmytro.lytovchenko@REDACTED Mon Aug 7 14:27:04 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Mon, 7 Aug 2017 14:27:04 +0200 Subject: [erlang-questions] List Question In-Reply-To: <1288613114.20170807221323@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> Message-ID: A string is a list of integers [$F, $T] (where $x is a code for the letter x and so on) A list of strings is a ? List of lists of integers. That is [ [$F, $T], [$S, $T] ] So to check whether a value is a list of lists you take its first element and check whether it's also a list function([X | _] = List) when is_list(X) -> got a nonempty list of strings! function([X | _]) when is_list(X) -> got a nonempty list of integers (one string) function([]) -> i am sorry, empty list is here there is no way to know having an empty list what it supposed to contain. So an empty list is an empty list 2017-08-07 14:13 GMT+02:00 Andrew McIntyre : > Hello All, > > A Newbie question, > > Can I tell the difference Between a List of Strings and a Single > String? > > eg the Value in a List may just be "FT" or it might be ["FT", "ST"] > > I do not know in advance if its a single string or a list of strings, > but want to behave differently - actually return the first element of > the list when there are multiple values. I have parsed a variable > hierarchical structure and the hierarchy can vary over instances. > > As strings are lists "is_list" returns true for both. Is that soluble > or do I need to change the data structure to indicate it. Sorry if > this has obvious answer, but new to erlang. > > Thanks > > -- > Best regards, > Andrew McIntyre mailto:andrew@REDACTED > > sent from a real computer > > R&D Director > Medical-Objects > 4/102 Wises Road > MAROOCHYDORE Q 4558 > AUSTRALIA > > www.medical-objects.com.au > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Mon Aug 7 14:53:11 2017 From: zxq9@REDACTED (zxq9) Date: Mon, 07 Aug 2017 21:53:11 +0900 Subject: [erlang-questions] List Question In-Reply-To: <654058386.20170807222931@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> Message-ID: <1627741.xJ36y4QYPI@changa> On 2017?08?07? ??? 22:29:31 you wrote: > Hello zxq9, > > Thanks, Unfortunately I do not know the value of the string that will > be there. Its an extensible hierarchy that can be several lists deep - > or not. Might need to revise the data structure In this case it can be useful to consider a way of tagging values. Imagine we want to represent a directory tree structure and have a descent-first traversal function recurse over it while creating the tree. We have two things that can happen, there is a flat list of new directories that need to be created, and there is the possibility that the tree depth extends deeper at each node. The naive version would look like what you have: ["top_dir_1", "top_dir_2", ["next_level_1", "next_level_2"]] This leaves a bit to be desired, not only because of the problem you have pointed out that makes it difficult to know what is deep and what is shallow, but also because we don't really have a good way to represent a full tree (what would be the name of a directory containing other directories?). So consider instead something like this: [{"top_dir_1", []}, {"top_dir_2", []}, {"top_dir_3", [{"next_level_1", []}, {"next_level_2", []}]}] Now we have a representation of each directory's name AND its contents. We can traverse this laterally AND in depth without any ambiguity or need for carrying around a record of where we have been (by using depth recursion and tail-call recursion): make_tree([{Dir, Contents} | Rest]) -> ok = case filelib:is_dir(Dir) of true -> ok; false -> ok = log(info, "Creating dir: ~p", [Dir]), file:make_dir(Dir) end, ok = file:set_cwd(Dir), ok = make_tree(Contents), ok = file:set_cwd(".."), make_tree(Rest); make_tree([]) -> ok. Not so bad. In your case we could represent things perhaps a bit better by separating the types and tagging them. Instead of just "FT" and whatever other string labels you might want, you could either use atoms (totally unambiguous) or tuples as we have in the example able (also totally unambiguous). I prefer tuples, though, because they are easier to read. [{value, "foo"}, {tree, [{value, "bar"}, {value, "foo"}]}, {value, "baz"}] So then we do something like: traverse([{value, Value} | Rest]) -> ok = do_thing(Value), traverse(Rest); traverse([{tree, Contents} | Rest]) -> ok = traverse(Contents), traverse(Rest); traverse([]) -> ok. Anyway, don't be afraid of varying your value types to say exactly what you mean. If your strings like "FT" only had meaning within your system consider NOT USING STRINGS, and using atoms instead. That makes it even easier: [foo, bar, [foo, bar], foo] So then we can do: traverse([foo | Rest]) -> ok = do_foo(), traverse(Rest); traverse([bar | Rest]) -> ok = do_bar(), traverse(Rest); traverse([Value | Rest]) when is_list(Value) -> ok = traverse(Value), traverse(Rest); traverse([]) -> ok. And of course, you can not use a guard if you want to match on a list shape in the listy clause there, but that is a minor detail. The point is to make your data types MEAN SOMETHING REASONABLE within your system. Use atoms when your values are meaningful only within your system. Strings are for the birds. -Craig From Oliver.Korpilla@REDACTED Mon Aug 7 14:57:28 2017 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Mon, 7 Aug 2017 14:57:28 +0200 Subject: [erlang-questions] List Question In-Reply-To: <1288613114.20170807221323@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> Message-ID: Hello. I would have done something like this: is_string(Input) when is_list(Input) -> lists:all(fun erlang:is_integer/1, Input); is_string(_Input) -> false. ? This works fine for regular strings (lists of integers). Erlang, however, often uses a construct called iolist for performant formatting of output. And these can contain lists of strings or list of lists in arbitrary depth, also not requiring them to be proper lists (they need not end with the empty list []). There more work would be required. Cheers, Oliver Gesendet:?Montag, 07. August 2017 um 14:13 Uhr Von:?"Andrew McIntyre" An:?"Erlang-Questions Questions" Betreff:?[erlang-questions] List Question Hello All, A Newbie question, Can I tell the difference Between a List of Strings and a Single String? eg the Value in a List may just be "FT" or it might be ["FT", "ST"] I do not know in advance if its a single string or a list of strings, but want to behave differently - actually return the first element of the list when there are multiple values. I have parsed a variable hierarchical structure and the hierarchy can vary over instances. As strings are lists "is_list" returns true for both. Is that soluble or do I need to change the data structure to indicate it. Sorry if this has obvious answer, but new to erlang. Thanks -- Best regards, Andrew McIntyre mailto:andrew@REDACTED sent from a real computer R&D Director Medical-Objects 4/102 Wises Road MAROOCHYDORE Q 4558 AUSTRALIA www.medical-objects.com.au[http://www.medical-objects.com.au] _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] From andrew@REDACTED Mon Aug 7 15:46:52 2017 From: andrew@REDACTED (Andrew McIntyre) Date: Mon, 7 Aug 2017 23:46:52 +1000 Subject: [erlang-questions] List Question In-Reply-To: <1627741.xJ36y4QYPI@changa> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> Message-ID: <978091051.20170807234652@Medical-Objects.com.au> Hello Craig, Thanks for your help. I am trying to store the data as efficiently as possible. Its HL7 natively and this is my test: OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F |~^& are delimiters. The hierarchy is only so deep and using lists of lists to provide a tree like way to access the data eg Field 3, repeat 1 component 2 subcomponent1 Parsed it looks like this: [["OBX","17", ["FT","TEST"], [["8265-1",[],["LN","SUBCOMP"]]], [[["1","2","3","4"]]], "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] As the format evolves over time the hierarchy can be extended, but older clients can still read the value they are expecting if they follow the rules, like reading the first value in the list when you only expect one value to be there. Currently a typical system might have 12 million of these records so want to keep format as small as possible in the erlang format, hence reluctant to tag 2 much, but know how to get value of interest. Maybe that is my non erlang background showing up? Traversing 4 small lists by index should be fast?? I guess I could save strings as binary in the lists then is_binary should work?? Is that the case. I gather on 64bit system especially binary is more space efficient. Monday, August 7, 2017, 10:53:11 PM, you wrote: z> On 2017?08?07? ??? 22:29:31 you wrote: >> Hello zxq9, >> >> Thanks, Unfortunately I do not know the value of the string that will >> be there. Its an extensible hierarchy that can be several lists deep - >> or not. Might need to revise the data structure z> In this case it can be useful to consider a way of tagging values. z> Imagine we want to represent a directory tree structure and have a z> descent-first traversal function recurse over it while creating the z> tree. We have two things that can happen, there is a flat list of z> new directories that need to be created, and there is the z> possibility that the tree depth extends deeper at each node. z> The naive version would look like what you have: z> ["top_dir_1", z> "top_dir_2", z> ["next_level_1", z> "next_level_2"]] z> This leaves a bit to be desired, not only because of the problem z> you have pointed out that makes it difficult to know what is deep z> and what is shallow, but also because we don't really have a good z> way to represent a full tree (what would be the name of a directory containing other directories?). z> So consider instead something like this: z> [{"top_dir_1", []}, z> {"top_dir_2", []}, z> {"top_dir_3", z> [{"next_level_1", []}, z> {"next_level_2", []}]}] z> Now we have a representation of each directory's name AND its contents. z> We can traverse this laterally AND in depth without any ambiguity z> or need for carrying around a record of where we have been (by z> using depth recursion and tail-call recursion): z> make_tree([{Dir, Contents} | Rest]) -> z> ok = z> case filelib:is_dir(Dir) of z> true -> z> ok; z> false -> z> ok = log(info, "Creating dir: ~p", [Dir]), z> file:make_dir(Dir) z> end, z> ok = file:set_cwd(Dir), z> ok = make_tree(Contents), z> ok = file:set_cwd(".."), z> make_tree(Rest); make_tree([]) ->> z> ok. z> Not so bad. z> In your case we could represent things perhaps a bit better by z> separating the types and tagging them. Instead of just "FT" and z> whatever other string labels you might want, you could either use z> atoms (totally unambiguous) or tuples as we have in the example z> able (also totally unambiguous). I prefer tuples, though, because they are easier to read. z> [{value, "foo"}, z> {tree, z> [{value, "bar"}, z> {value, "foo"}]}, z> {value, "baz"}] z> So then we do something like: z> traverse([{value, Value} | Rest]) -> z> ok = do_thing(Value), z> traverse(Rest); z> traverse([{tree, Contents} | Rest]) -> z> ok = traverse(Contents), z> traverse(Rest); traverse([]) ->> z> ok. z> Anyway, don't be afraid of varying your value types to say exactly z> what you mean. If your strings like "FT" only had meaning within z> your system consider NOT USING STRINGS, and using atoms instead. That makes it even easier: z> [foo, z> bar, z> [foo, z> bar], z> foo] z> So then we can do: z> traverse([foo | Rest]) -> z> ok = do_foo(), z> traverse(Rest); z> traverse([bar | Rest]) -> z> ok = do_bar(), z> traverse(Rest); z> traverse([Value | Rest]) when is_list(Value) -> z> ok = traverse(Value), z> traverse(Rest); traverse([]) ->> z> ok. z> And of course, you can not use a guard if you want to match on a z> list shape in the listy clause there, but that is a minor detail. z> The point is to make your data types MEAN SOMETHING REASONABLE z> within your system. Use atoms when your values are meaningful only z> within your system. Strings are for the birds. z> -Craig z> _______________________________________________ z> erlang-questions mailing list z> erlang-questions@REDACTED z> http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Andrew mailto:andrew@REDACTED sent from a real computer From zxq9@REDACTED Mon Aug 7 16:39:05 2017 From: zxq9@REDACTED (zxq9) Date: Mon, 07 Aug 2017 23:39:05 +0900 Subject: [erlang-questions] List Question In-Reply-To: <978091051.20170807234652@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> Message-ID: <1629505.EyzGccMyjN@changa> On 2017?08?07? ??? 23:46:52 you wrote: > Hello Craig, > > Thanks for your help. > > I am trying to store the data as efficiently as possible. Its HL7 > natively and this is my test: > > OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F > > |~^& are delimiters. The hierarchy is only so deep and using lists of > lists to provide a tree like way to access the data eg Field 3, repeat > 1 component 2 subcomponent1 > > Parsed it looks like this: > > [["OBX","17", > ["FT","TEST"], > [["8265-1",[],["LN","SUBCOMP"]]], > [[["1","2","3","4"]]], > "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] > > As the format evolves over time the hierarchy can be extended, but > older clients can still read the value they are expecting if they > follow the rules, like reading the first value in the list when you > only expect one value to be there. > > Currently a typical system might have 12 million of these records so > want to keep format as small as possible in the erlang format, hence > reluctant to tag 2 much, but know how to get value of interest. Maybe > that is my non erlang background showing up? Traversing 4 small lists > by index should be fast?? > > I guess I could save strings as binary in the lists then is_binary > should work?? Is that the case. I gather on 64bit system especially > binary is more space efficient. We would really want to know a bit about the data's natural semantics before leaping to conclusions regarding "most efficient" or especially "best" internal representation. On the note of "internal representation"... do you need 12 million records in memory at a given time, or are these things you can store as strings and then pull the ones you want as you go? Do you need random access to records? To be able to query them? Are you performing aggregate operations over them? All of these might change the answer of "what is best". For example, if you need a few tens of millions of records in memory at a given time AND the short strings like "FT" and "TEST" are always members of a smallish, finite set you know beforehand then atoms will be much more space efficient if that is your main concern. Atoms are tiny in memory (essentially like using integer constants in a C program), but they serialize back to strings in storage so that can change things. For example, the atom 'test' (or 'TEST') is smaller in memory than either the string "TEST" or the binary <<"TEST">>. In any case, just to get the code right first so you have a platform to change things around I would recommend TOTALLY FORGETTING ABOUT THIS to start with and jump straight to tagged pairs. This will keep your semantics straight and dramatically reduce the mental overhead of writing traversal functions. You can switch that stuff around later -- but first you need things to work at least partially before you can start measuring stuff. For example: > [["OBX","17", > ["FT","TEST"], > [["8265-1",[],["LN","SUBCOMP"]]], > [[["1","2","3","4"]]], > "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] I would parse that to something like this: [{foo, "OBX"}, {bar, 17}, {baz, {"FT", "TEST"}}, [[{biz, "8265-1"} {boz, []}, {ballz, {"LN", "SUBCOMP"}}]], [[{bonz, [1, 2, 3, 4]}]], {coz, "\\H\\Spot Image 2\\N\\"}, {fuzz, "F"}] The leading tag should always disambiguate the MEANING of the element you are encountering in your lists. Here I used junk terms, but I think you get the idea. Whatever atom labels you use should be unambiguous and mean something to the humans reading the code (or crash dumps). The overhead here VS the list version is the atoms and enclosing tuple (a few bytes for both) -- which is negligible compared to the cost of using actual strings directly in your code. Remember, an atom is basically an alias for a constant, so don't fret over it too much -- just don't start doing insane things like dynamically generating an arbitrary number of unique atoms! But again, remember, we don't care how big this is in memory yet. What we care about is that it is easy to traverse and know what you mean when you do so, and also that it is easy to unambiguously cast the data from an old format to a new one (which is inherently easier with tagged data than with semantically amibugous lists of lists). Obviously giving further advice would be a bit easier if I knew more about the meaning of the data itself, but anyway, the primary concern here is getting the logic right first. You can crunch things down later -- and most of the time that usually has much less to do with finding a "small" internal representation for those moments while you are actively handling some bits of data in memory and a LOT more to do with how you are going to index serialized data on disk (ETS/DETS/Mnesia may be a good route for that, or Postgres, or maps, whatever -- depending on what you're up to). Hopefully I explained more than confused. -Craig From juanjo@REDACTED Mon Aug 7 17:32:23 2017 From: juanjo@REDACTED (Juan Jose Comellas) Date: Mon, 7 Aug 2017 12:32:23 -0300 Subject: [erlang-questions] List Question In-Reply-To: <978091051.20170807234652@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> Message-ID: Andrew, if you want to store the data in a format that is as compact as possible, I'd recommend storing the HL7 message itself as a binary and parsing on demand. If you want to store the data pre-parsed, then I would store them as list of segments where each segment is represented by a nested tuple. That way you can reference the fields, components, etc., by their index in an O(1) operation, and you can still easily add or remove segments from a message. What I'm describing is similar to the intermediate format used by an HL7 parser (https://github.com/jcomellas/ex_hl7) I wrote for Elixir. You could probably use it as inspiration for what you need. I had also created another parser in Erlang (https://github.com/jcomellas/ehl7) that maps the segments to records, but part of it is in C using NIFs. Let me know if you have any other doubts. On Mon, Aug 7, 2017 at 10:46 AM, Andrew McIntyre < andrew@REDACTED> wrote: > Hello Craig, > > Thanks for your help. > > I am trying to store the data as efficiently as possible. Its HL7 > natively and this is my test: > > OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F > > |~^& are delimiters. The hierarchy is only so deep and using lists of > lists to provide a tree like way to access the data eg Field 3, repeat > 1 component 2 subcomponent1 > > Parsed it looks like this: > > [["OBX","17", > ["FT","TEST"], > [["8265-1",[],["LN","SUBCOMP"]]], > [[["1","2","3","4"]]], > "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] > > As the format evolves over time the hierarchy can be extended, but > older clients can still read the value they are expecting if they > follow the rules, like reading the first value in the list when you > only expect one value to be there. > > Currently a typical system might have 12 million of these records so > want to keep format as small as possible in the erlang format, hence > reluctant to tag 2 much, but know how to get value of interest. Maybe > that is my non erlang background showing up? Traversing 4 small lists > by index should be fast?? > > I guess I could save strings as binary in the lists then is_binary > should work?? Is that the case. I gather on 64bit system especially > binary is more space efficient. > > Monday, August 7, 2017, 10:53:11 PM, you wrote: > > z> On 2017?08?07? ??? 22:29:31 you wrote: > >> Hello zxq9, > >> > >> Thanks, Unfortunately I do not know the value of the string that will > >> be there. Its an extensible hierarchy that can be several lists deep - > >> or not. Might need to revise the data structure > > z> In this case it can be useful to consider a way of tagging values. > > z> Imagine we want to represent a directory tree structure and have a > z> descent-first traversal function recurse over it while creating the > z> tree. We have two things that can happen, there is a flat list of > z> new directories that need to be created, and there is the > z> possibility that the tree depth extends deeper at each node. > > z> The naive version would look like what you have: > > z> ["top_dir_1", > z> "top_dir_2", > z> ["next_level_1", > z> "next_level_2"]] > > z> This leaves a bit to be desired, not only because of the problem > z> you have pointed out that makes it difficult to know what is deep > z> and what is shallow, but also because we don't really have a good > z> way to represent a full tree (what would be the name of a directory > containing other directories?). > > z> So consider instead something like this: > > z> [{"top_dir_1", []}, > z> {"top_dir_2", []}, > z> {"top_dir_3", > z> [{"next_level_1", []}, > z> {"next_level_2", []}]}] > > z> Now we have a representation of each directory's name AND its contents. > > z> We can traverse this laterally AND in depth without any ambiguity > z> or need for carrying around a record of where we have been (by > z> using depth recursion and tail-call recursion): > > > z> make_tree([{Dir, Contents} | Rest]) -> > z> ok = > z> case filelib:is_dir(Dir) of > z> true -> > z> ok; > z> false -> > z> ok = log(info, "Creating dir: ~p", [Dir]), > z> file:make_dir(Dir) > z> end, > z> ok = file:set_cwd(Dir), > z> ok = make_tree(Contents), > z> ok = file:set_cwd(".."), > z> make_tree(Rest); > make_tree([]) ->> > z> ok. > > > z> Not so bad. > > z> In your case we could represent things perhaps a bit better by > z> separating the types and tagging them. Instead of just "FT" and > z> whatever other string labels you might want, you could either use > z> atoms (totally unambiguous) or tuples as we have in the example > z> able (also totally unambiguous). I prefer tuples, though, because they > are easier to read. > > z> [{value, "foo"}, > z> {tree, > z> [{value, "bar"}, > z> {value, "foo"}]}, > z> {value, "baz"}] > > > z> So then we do something like: > > > z> traverse([{value, Value} | Rest]) -> > z> ok = do_thing(Value), > z> traverse(Rest); > z> traverse([{tree, Contents} | Rest]) -> > z> ok = traverse(Contents), > z> traverse(Rest); > traverse([]) ->> > z> ok. > > > z> Anyway, don't be afraid of varying your value types to say exactly > z> what you mean. If your strings like "FT" only had meaning within > z> your system consider NOT USING STRINGS, and using atoms instead. That > makes it even easier: > > > z> [foo, > z> bar, > z> [foo, > z> bar], > z> foo] > > > z> So then we can do: > > > z> traverse([foo | Rest]) -> > z> ok = do_foo(), > z> traverse(Rest); > z> traverse([bar | Rest]) -> > z> ok = do_bar(), > z> traverse(Rest); > z> traverse([Value | Rest]) when is_list(Value) -> > z> ok = traverse(Value), > z> traverse(Rest); > traverse([]) ->> > z> ok. > > > z> And of course, you can not use a guard if you want to match on a > z> list shape in the listy clause there, but that is a minor detail. > z> The point is to make your data types MEAN SOMETHING REASONABLE > z> within your system. Use atoms when your values are meaningful only > z> within your system. Strings are for the birds. > > z> -Craig > z> _______________________________________________ > z> erlang-questions mailing list > z> erlang-questions@REDACTED > z> http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Best regards, > Andrew mailto:andrew@REDACTED > > sent from a real computer > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Aug 7 23:18:27 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 8 Aug 2017 09:18:27 +1200 Subject: [erlang-questions] Should erlang have macro FUNCTION_ARGS ? In-Reply-To: References: Message-ID: On 7/08/17 8:22 PM, Zhongzheng Liu wrote: >> I don't think this *CAN* work in the presence of maps. > I think it can work with temporary variable. > > foo(X, [a,b,c], 42) -> > ?FUNCTION_ARGS. > > will become: > > foo(X=__TEMP_1__, [a,b,c]=__TEMP_2__, #{ b := Y }=__TEMP_3__) -> > [__TEMP_1__, __TEMP_2__, __TEMP_3__] . > > It will work with maps. OK, but now you are asking for something other than a macro. One key thing about a macro is that expanding a macro invocation has no effect whatsoever on the token sequence outside that invocation. It seems to me that if you really want this feature, you should use the machinery that already exists for the purpose. That is, you should write a parse transform. Parse transforms can pretty much rewrite anything to anything, and you don't have to wait for anyone else to change the Erlang system, you can just plug in your own parse transform any time you want. From ok@REDACTED Mon Aug 7 23:26:12 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 8 Aug 2017 09:26:12 +1200 Subject: [erlang-questions] List Question In-Reply-To: <1288613114.20170807221323@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> Message-ID: <7c6bb6ee-a79f-8d90-6cd5-32461e58bfef@cs.otago.ac.nz> On 8/08/17 12:13 AM, Andrew McIntyre wrote: > Hello All, > > A Newbie question, > > Can I tell the difference Between a List of Strings and a Single > String? Not if it is an empty list. If it is OK to treat an empty list of strings as if it were a list containing one empty string, you can use this: classify([[_|_]|_]) -> list; classify([[]|_] ) -> list; classify([C|_] ) when is_integer(C) -> string; classify([] ) -> string. % GUESS > > I do not know in advance if its a single string or a list of strings, > but want to behave differently In this case you have a poor data structure choice. You should be passing {string,S} or {string_list,L} or something like that. Or you might want to use binaries instead of character lists. - actually return the first element of > the list when there are multiple values. In this particular case, it sounds as though first_string([S|]) when is_list(S) -> S; first_string(S) when is_list(S) -> S. might do. But seriously, one of the first rules in any Lispy language, like Lisp, Scheme, Pop-2, Pop-11, Prolog, or Erlang, is Don't Make Your Data Ambiguous. It's OK to have a data structure that could, in a different context, be interpreted differently, but if you pass something to a function, it is up to you to make sure the function gets all the information it needs. From ok@REDACTED Tue Aug 8 00:24:06 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 8 Aug 2017 10:24:06 +1200 Subject: [erlang-questions] List Question In-Reply-To: <978091051.20170807234652@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> Message-ID: <4e20fc7c-0691-2a9e-d2b2-5ef5757ac4ca@cs.otago.ac.nz> On 8/08/17 1:46 AM, Andrew McIntyre wrote: > Hello Craig, > > Thanks for your help. > > I am trying to store the data as efficiently as possible. What do you mean "efficiently"? Do you mean "in the least SPACE possible"? In that case you should consider using binaries instead of strings. Do you mean "taking the least TIME possible for operations"? In that case, you need to tag your data so that the code that carries out the operations KNOWS right away what it is dealing with. > Its HL7 > natively and this is my test: hl7.org lists a lot of standards; being unfamiliar with them all I don't know which you are referring to. And you have to sign up in order to read any of them, and I dislike signing up for things when I don't know the consequences of signing up. > > OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F > > |~^& are delimiters. I take it that "|" is the first level delimiter (why the heck are they not using the Information Separator characters ASCII has specifically for this purpose?), ~ is the second level, ^ is the third level, and & is the fourth level. Ah, wait: ::= {"\r" }... ::= {"|" }... ::= {"^" }... ::= {"&" }... ::= {"~" } ::= {[^\r|^&~\\]|\.]}... The first field of a segment appears to be a message type dentifier. One data structure for this would be -type message() = list(segment()). -type segment() = list(field()). -type field() = list(subfield()). -type subfield() = list(subsubfield()) -type subsubfield() = list(repeating()). -type repeating() = string(). [[[["OBX"]]], [[["17"]]], [[["FT", "TEST"]]], [[["8265-1"]], [[[]]], [["LN"], ["SUBCOMP"]]], [[["1"], ["2"], ["3"], ["4"]]], [[["\\H\\Spot Image 2\\N\\"]]], [[[""]]], [[[""]]], [[[""]]], [[[""]]], [[[""]]], [[["F"]]]] This is completely unambiguous. Now let's suppose we use a slightly different data structure. -type hier() = string() | {many,list(hier())}. We get {many,["OBX","17",{many,["FT","TEST"]},{many,["8265-1","",{many,["LN","SUBCOMP"]}]},{many,["1","2","3","4"]},"\\H\\Spot Image 2\\N\\","","","","","","F"]} Note that this is unambiguous but pays the price of a {many,_} wrapper only when there is more than one subpart, so the relative overhead is not so large. first({many,[H|_]) -> first(H); first(X) -> X. > Currently a typical system might have 12 million of these records so > want to keep format as small as possible in the erlang format, The way to do that would be to compress each segment, keep the entire segment as a binary, and only decompress and parse it when you need to look at that particular segment. A rough estimate of the size of this particular example is 108 words, which is actually less than the string took. Let's say that an average message takes 200 words in the {many,_} | string() form. 32-bit: 9.6 GB, not going to fit. 64-bit: 19.2 GB, this machine has 16 GB, oh dear. Even if we assume the average segment is 100 bytes, 12 million of those is 1.2 GB (excluding the cost of a binary for each segment. That's definitely doable. So we're looking at three options, I think: (1) Keeping segments as binaries, and parsing them every time you need to look inside. (2) Keeping segments as Erlang data structures in a Mnesia table (that is, on disc). (3) Streaming messages through an Erlang program instead of storing them all in memory at once. hence > reluctant to tag 2 much, but know how to get value of interest. Maybe > that is my non erlang background showing up? Traversing 4 small lists > by index should be fast?? > > I guess I could save strings as binary in the lists then is_binary > should work?? Is that the case. I gather on 64bit system especially > binary is more space efficient. > > Monday, August 7, 2017, 10:53:11 PM, you wrote: > > z> On 2017?08?07? ??? 22:29:31 you wrote: >>> Hello zxq9, >>> >>> Thanks, Unfortunately I do not know the value of the string that will >>> be there. Its an extensible hierarchy that can be several lists deep - >>> or not. Might need to revise the data structure > > z> In this case it can be useful to consider a way of tagging values. > > z> Imagine we want to represent a directory tree structure and have a > z> descent-first traversal function recurse over it while creating the > z> tree. We have two things that can happen, there is a flat list of > z> new directories that need to be created, and there is the > z> possibility that the tree depth extends deeper at each node. > > z> The naive version would look like what you have: > > z> ["top_dir_1", > z> "top_dir_2", > z> ["next_level_1", > z> "next_level_2"]] > > z> This leaves a bit to be desired, not only because of the problem > z> you have pointed out that makes it difficult to know what is deep > z> and what is shallow, but also because we don't really have a good > z> way to represent a full tree (what would be the name of a directory containing other directories?). > > z> So consider instead something like this: > > z> [{"top_dir_1", []}, > z> {"top_dir_2", []}, > z> {"top_dir_3", > z> [{"next_level_1", []}, > z> {"next_level_2", []}]}] > > z> Now we have a representation of each directory's name AND its contents. > > z> We can traverse this laterally AND in depth without any ambiguity > z> or need for carrying around a record of where we have been (by > z> using depth recursion and tail-call recursion): > > > z> make_tree([{Dir, Contents} | Rest]) -> > z> ok = > z> case filelib:is_dir(Dir) of > z> true -> > z> ok; > z> false -> > z> ok = log(info, "Creating dir: ~p", [Dir]), > z> file:make_dir(Dir) > z> end, > z> ok = file:set_cwd(Dir), > z> ok = make_tree(Contents), > z> ok = file:set_cwd(".."), > z> make_tree(Rest); > make_tree([]) ->> > z> ok. > > > z> Not so bad. > > z> In your case we could represent things perhaps a bit better by > z> separating the types and tagging them. Instead of just "FT" and > z> whatever other string labels you might want, you could either use > z> atoms (totally unambiguous) or tuples as we have in the example > z> able (also totally unambiguous). I prefer tuples, though, because they are easier to read. > > z> [{value, "foo"}, > z> {tree, > z> [{value, "bar"}, > z> {value, "foo"}]}, > z> {value, "baz"}] > > > z> So then we do something like: > > > z> traverse([{value, Value} | Rest]) -> > z> ok = do_thing(Value), > z> traverse(Rest); > z> traverse([{tree, Contents} | Rest]) -> > z> ok = traverse(Contents), > z> traverse(Rest); > traverse([]) ->> > z> ok. > > > z> Anyway, don't be afraid of varying your value types to say exactly > z> what you mean. If your strings like "FT" only had meaning within > z> your system consider NOT USING STRINGS, and using atoms instead. That makes it even easier: > > > z> [foo, > z> bar, > z> [foo, > z> bar], > z> foo] > > > z> So then we can do: > > > z> traverse([foo | Rest]) -> > z> ok = do_foo(), > z> traverse(Rest); > z> traverse([bar | Rest]) -> > z> ok = do_bar(), > z> traverse(Rest); > z> traverse([Value | Rest]) when is_list(Value) -> > z> ok = traverse(Value), > z> traverse(Rest); > traverse([]) ->> > z> ok. > > > z> And of course, you can not use a guard if you want to match on a > z> list shape in the listy clause there, but that is a minor detail. > z> The point is to make your data types MEAN SOMETHING REASONABLE > z> within your system. Use atoms when your values are meaningful only > z> within your system. Strings are for the birds. > > z> -Craig > z> _______________________________________________ > z> erlang-questions mailing list > z> erlang-questions@REDACTED > z> http://erlang.org/mailman/listinfo/erlang-questions > > > From publicityifl@REDACTED Mon Aug 7 20:35:09 2017 From: publicityifl@REDACTED (publicityifl@REDACTED) Date: Mon, 07 Aug 2017 18:35:09 +0000 Subject: [erlang-questions] Extended deadline: IFL 2017 (29th Symposium on Implementation and Application of Functional Languages) Message-ID: <001a114f206a45acab05562e1be4@google.com>
Hello,

Please, find below the third call for papers for IFL 2017. Note that some
of the deadlines have been extended. Details are given below.
Please forward these to anyone you think may be interested.
Apologies for any duplicates you may receive.

best regards,
Jurriaan Hage
Publicity Chair of IFL

---

IFL 2017 - CALL FOR PAPERS: DEADLINES EXTENDED
==============================================

29th SYMPOSIUM ON IMPLEMENTATION AND APPLICATION OF FUNCTIONAL LANGUAGES
=======================================================================

University of Bristol, UK

In cooperation with ACM SIGPLAN

Wednesday 30 August - Friday 1 September, 2017

http://iflconference.org/

Scope
-----

The goal of the IFL symposia is to bring together researchers actively engaged
in the implementation and application of functional and function-based
programming languages. IFL 2017 will be a venue for researchers to present and
discuss new ideas and concepts, work in progress, and publication-ripe results
related to the implementation and application of functional languages and
function-based programming.

Peer-review
-----------

Following the IFL tradition, IFL 2017 will use a post-symposium review process
to produce the formal proceedings. All participants of IFL 2017 are invited to
submit either a draft paper or an extended abstract describing work to be
presented at the symposium. At no time may work submitted to IFL be
simultaneously submitted to other venues; submissions must adhere to ACM

SIGPLAN's republication policy:

http://www.sigplan.org/Resources/Policies/Republication

The submissions will be screened by the program committee chair to make sure
they are within the scope of IFL, and will appear in the draft proceedings
distributed at the symposium. Submissions appearing in the draft proceedings
are not peer-reviewed publications. Hence, publications that appear only in the
draft proceedings are not subject to the ACM SIGPLAN republication policy.

After the symposium, authors will be given the opportunity to incorporate the
feedback from discussions at the symposium and will be invited to submit a
revised full article for the formal review process. From the revised
submissions, the program committee will select papers for the formal
proceedings considering their correctness, novelty, originality, relevance,
significance, and clarity. The formal proceedings will appear in the
International Conference Proceedings Series of the ACM Digital Library.

Important dates
---------------

Note that the original deadlines for submission and registration have
been extended.

|????????????????????? |?????????????????????????????????????????????????????????? |
| ----------------------| -----------------------------------------------------------|
| Thu 17 August?? 2017 | Submission deadline draft papers????????????????????????? |
| Fri 18 August?? 2017 | Notification of acceptance for presentation?????????????? |
| Mon 21 August?? 2017 | Early registration deadline?????????????????????????????? |
| Mon 21 August?? 2017 | Submission deadline for pre-symposium proceedings???????? |
| Fri 25 August?? 2017 | Late registration deadline??????????????????????????????? |
| Wed 30 August?? 2017 - Fri 1 September 2017 | IFL Symposium????????????????????? |
| Mon? 4 December 2017 | Submission deadline for post-symposium proceedings??????? |
| Wed 31 January? 2018 | Notification of acceptance for post-symposium proceedings |
| Mon 12 March??? 2018 | Camera-ready version for post-symposium proceedings?????? |

Submission details
------------------

Prospective authors are encouraged to submit papers or extended abstracts to be
published in the draft proceedings and to present them at the symposium. All
contributions must be written in English. Papers must use the new ACM two
columns conference format, which can be found at:

http://www.acm.org/publications/proceedings-template

For the pre-symposium proceedings we adopt a 'weak' page limit of 12 pages. For
the post-symposium proceedings the page limit of 12 pages is firm.

Authors submit through EasyChair:

https://easychair.org/conferences/?conf=ifl2017

Topics
------

IFL welcomes submissions describing practical and theoretical work as well as
submissions describing applications and tools in the context of functional
programming. If you are not sure whether your work is appropriate for IFL 2017,
please contact the PC chair at nicolas.wu@REDACTED. Topics of interest include, but are not limited to:

- language concepts
- type systems, type checking, type inferencing
- compilation techniques
- staged compilation
- run-time function specialization
- run-time code generation
- partial evaluation
- (abstract) interpretation
- metaprogramming
- generic programming
- automatic program generation
- array processing
- concurrent/parallel programming
- concurrent/parallel program execution
- embedded systems
- web applications
- (embedded) domain specific languages
- security
- novel memory management techniques
- run-time profiling performance measurements
- debugging and tracing
- virtual/abstract machine architectures
- validation, verification of functional programs
- tools and programming techniques
- (industrial) applications

Peter Landin Prize
------------------

The Peter Landin Prize is awarded to the best paper presented at the symposium
every year. The honored article is selected by the program committee based on
the submissions received for the formal review process. The prize carries a
cash award equivalent to 150 Euros.


Programme committee
-------------------

Chair: Nicolas Wu, University of Bristol, UK

- Kenichi Asai, Ochanomizu University, Japan
- Sandrine Blazy, University of Rennes 1, France
- Carlos Camarao, Universidade Federal de Minas Gerais, Brazil
- Stephen Dolan, University of Cambridge, UK
- Jurriaan Hage, Utrecht University, Netherlands
- Yukiyoshi Kameyama, University of Tsukuba, Japan
- Benjamin Lerner, Brown University, USA
- Bas Lijnse, Radboud University, Netherlands
- Garrett Morris, University of Kansas, USA
- Miguel Pagano, Universidad Nacional de C??rdoba, Argentina
- Tomas Petricek, Alan Turing Institute, UK
- Maciej Pir??g, University of Wroclaw, Poland
- Exequiel Rivas, Universidad Nacional de Rosario, Argentina
- Neil Sculthorpe, Nottingham Trent University, UK
- Melinda Toth, Eotvos Lorand University, Hungary
- Phil Trinder, Glasgow University, UK
- Kanae Tsushima, National Institute of Informatics, Japan
- Marcos Viera, Universidad de la Republica, Uruguay
- Meng Wang, University of Kent, UK

Venue
-----

The IFL 2017 will be held in association with the Department of
Computer Science, University of Bristol, UK. Bristol is located in
South West England, and can be easily reached from Bristol Airport.

See the website for more information on the venue.

 

powered by GSM. Free mail merge and email marketing software for Gmail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Tue Aug 8 10:00:59 2017 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 8 Aug 2017 10:00:59 +0200 Subject: [erlang-questions] List Question In-Reply-To: <978091051.20170807234652@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> Message-ID: Hello, I'm going to go way off topic here and not answer your specific question about lists ... Your last mail had the information I need - you're trying to parse HL7. I have a few comments. 1) Your original question did not bother to mention what problem you were trying to solve - You asked about a sub-problem that you encountered when trying to solve your principle problem (principle problem = parse HL7) (sub-problem = differentiate lists) 2) It's *always* a good idea to ask questions about the principle problem first !!!! I didn't know what HL7 was - my immediate thought was 'I wonder if anybody has written an *proper* HL7 parser in Erlang' - by proper I mean "has expended a significant amount of thought on writing a parser" Google is your friend - It told me what HL7 was (I hadn't a clue here - "never heard of it") and it turned up a parser in elixir https://github.com/jcomellas/ex_hl7 >From the quality of the documentation I assume this is a *proper* implementation. Now elixir compiles to .beam files and can be called from Erlang - which raises another sub problem "how do I compile the elixir code and call it from Erlang" and begs the question "is this effort worthwhile" Given that a parser for HL7 exists in elixir it might be sensible to use it "off the shelf" I have a feeling that elixir folks are good at reusing erlang code - but that reuse in the opposite direction is less easy. The last time I fiddled a bit (yesterday as it happened) - it turned out to be less than blindingly obvious how to call other than trivial elixir code from erlang. I was also wondering about cross-compilation. Has anybody written something that turns erlang code into elixir source code or vice. versa. Cheers /Joe On Mon, Aug 7, 2017 at 3:46 PM, Andrew McIntyre wrote: > Hello Craig, > > Thanks for your help. > > I am trying to store the data as efficiently as possible. Its HL7 > natively and this is my test: > > OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F > > |~^& are delimiters. The hierarchy is only so deep and using lists of > lists to provide a tree like way to access the data eg Field 3, repeat > 1 component 2 subcomponent1 > > Parsed it looks like this: > > [["OBX","17", > ["FT","TEST"], > [["8265-1",[],["LN","SUBCOMP"]]], > [[["1","2","3","4"]]], > "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] > > As the format evolves over time the hierarchy can be extended, but > older clients can still read the value they are expecting if they > follow the rules, like reading the first value in the list when you > only expect one value to be there. > > Currently a typical system might have 12 million of these records so > want to keep format as small as possible in the erlang format, hence > reluctant to tag 2 much, but know how to get value of interest. Maybe > that is my non erlang background showing up? Traversing 4 small lists > by index should be fast?? > > I guess I could save strings as binary in the lists then is_binary > should work?? Is that the case. I gather on 64bit system especially > binary is more space efficient. > > Monday, August 7, 2017, 10:53:11 PM, you wrote: > > z> On 2017?08?07? ??? 22:29:31 you wrote: >>> Hello zxq9, >>> >>> Thanks, Unfortunately I do not know the value of the string that will >>> be there. Its an extensible hierarchy that can be several lists deep - >>> or not. Might need to revise the data structure > > z> In this case it can be useful to consider a way of tagging values. > > z> Imagine we want to represent a directory tree structure and have a > z> descent-first traversal function recurse over it while creating the > z> tree. We have two things that can happen, there is a flat list of > z> new directories that need to be created, and there is the > z> possibility that the tree depth extends deeper at each node. > > z> The naive version would look like what you have: > > z> ["top_dir_1", > z> "top_dir_2", > z> ["next_level_1", > z> "next_level_2"]] > > z> This leaves a bit to be desired, not only because of the problem > z> you have pointed out that makes it difficult to know what is deep > z> and what is shallow, but also because we don't really have a good > z> way to represent a full tree (what would be the name of a directory containing other directories?). > > z> So consider instead something like this: > > z> [{"top_dir_1", []}, > z> {"top_dir_2", []}, > z> {"top_dir_3", > z> [{"next_level_1", []}, > z> {"next_level_2", []}]}] > > z> Now we have a representation of each directory's name AND its contents. > > z> We can traverse this laterally AND in depth without any ambiguity > z> or need for carrying around a record of where we have been (by > z> using depth recursion and tail-call recursion): > > > z> make_tree([{Dir, Contents} | Rest]) -> > z> ok = > z> case filelib:is_dir(Dir) of > z> true -> > z> ok; > z> false -> > z> ok = log(info, "Creating dir: ~p", [Dir]), > z> file:make_dir(Dir) > z> end, > z> ok = file:set_cwd(Dir), > z> ok = make_tree(Contents), > z> ok = file:set_cwd(".."), > z> make_tree(Rest); > make_tree([]) ->> > z> ok. > > > z> Not so bad. > > z> In your case we could represent things perhaps a bit better by > z> separating the types and tagging them. Instead of just "FT" and > z> whatever other string labels you might want, you could either use > z> atoms (totally unambiguous) or tuples as we have in the example > z> able (also totally unambiguous). I prefer tuples, though, because they are easier to read. > > z> [{value, "foo"}, > z> {tree, > z> [{value, "bar"}, > z> {value, "foo"}]}, > z> {value, "baz"}] > > > z> So then we do something like: > > > z> traverse([{value, Value} | Rest]) -> > z> ok = do_thing(Value), > z> traverse(Rest); > z> traverse([{tree, Contents} | Rest]) -> > z> ok = traverse(Contents), > z> traverse(Rest); > traverse([]) ->> > z> ok. > > > z> Anyway, don't be afraid of varying your value types to say exactly > z> what you mean. If your strings like "FT" only had meaning within > z> your system consider NOT USING STRINGS, and using atoms instead. That makes it even easier: > > > z> [foo, > z> bar, > z> [foo, > z> bar], > z> foo] > > > z> So then we can do: > > > z> traverse([foo | Rest]) -> > z> ok = do_foo(), > z> traverse(Rest); > z> traverse([bar | Rest]) -> > z> ok = do_bar(), > z> traverse(Rest); > z> traverse([Value | Rest]) when is_list(Value) -> > z> ok = traverse(Value), > z> traverse(Rest); > traverse([]) ->> > z> ok. > > > z> And of course, you can not use a guard if you want to match on a > z> list shape in the listy clause there, but that is a minor detail. > z> The point is to make your data types MEAN SOMETHING REASONABLE > z> within your system. Use atoms when your values are meaningful only > z> within your system. Strings are for the birds. > > z> -Craig > z> _______________________________________________ > z> erlang-questions mailing list > z> erlang-questions@REDACTED > z> http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Best regards, > Andrew mailto:andrew@REDACTED > > sent from a real computer > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Tue Aug 8 10:48:35 2017 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 8 Aug 2017 01:48:35 -0700 Subject: [erlang-questions] List Question In-Reply-To: References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> Message-ID: <59897AE3.6040609@gmail.com> On 08/08/2017 01:00 AM, Joe Armstrong wrote: > The last time I fiddled a bit (yesterday as it happened) - it turned > out to be less than > blindingly obvious how to call other than trivial elixir code from erlang. > > I was also wondering about cross-compilation. Has anybody written > something that turns > erlang code into elixir source code or vice. versa. I created a script ex2erl (https://github.com/okeuday/reltool_util/blob/master/ex2erl) to convert a single Elixir module into multiple Erlang modules. Then it is easy to include Elixir source code into an Erlang project without creating a dependency on Elixir for all use of the Erlang project. The ex2erl script requires an Elixir installation alongside the Erlang installation and depends on the Elixir module having the debug_info in the beam output which is done by default. Best Regards, Michael From zerthurd@REDACTED Tue Aug 8 10:59:15 2017 From: zerthurd@REDACTED (Max Treskin) Date: Tue, 8 Aug 2017 01:59:15 -0700 Subject: [erlang-questions] Memory leak in SSL Message-ID: Hello, I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks. ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM. What should I do to eliminate this? Is it bug or intended behaviour? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From spylik@REDACTED Tue Aug 8 10:24:00 2017 From: spylik@REDACTED (Oleksii Semilietov) Date: Tue, 8 Aug 2017 11:24:00 +0300 Subject: [erlang-questions] List Question In-Reply-To: References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> Message-ID: I using parse_trans_pp from Ulf Wiger parse_trans lib ( https://github.com/uwiger/parse_trans) to get erlang code from elixir compiled modules. Common technique is compile elixir project than do escript ~/projects/parse_trans/ebin/parse_trans_pp.beam _build/dev/lib/some_project/ebin/module_name.beam > module_name.erl Output will be erang module. Do somebody have other ways? On 8 August 2017 at 11:00, Joe Armstrong wrote: > Hello, > > I'm going to go way off topic here and not answer your specific > question about lists ... > > Your last mail had the information I need - you're trying to parse HL7. > I have a few comments. > > 1) Your original question did not bother to mention what problem you > were trying to solve - > You asked about a sub-problem that you encountered when trying to > solve your principle > problem (principle problem = parse HL7) (sub-problem = differentiate > lists) > > 2) It's *always* a good idea to ask questions about the principle > problem first !!!! > > I didn't know what HL7 was - my immediate thought was > 'I wonder if anybody has written an *proper* HL7 parser in Erlang' - by > proper I mean "has expended a significant amount of thought on writing a > parser" > > Google is your friend - It told me what HL7 was (I hadn't a clue here > - "never heard of it") > and it turned up a parser in elixir > > https://github.com/jcomellas/ex_hl7 > > From the quality of the documentation I assume this is a *proper* > implementation. > > Now elixir compiles to .beam files and can be called from Erlang - > which raises another > sub problem "how do I compile the elixir code and call it from Erlang" > and begs the > question "is this effort worthwhile" > > Given that a parser for HL7 exists in elixir it might be sensible to > use it "off the shelf" > > I have a feeling that elixir folks are good at reusing erlang code - > but that reuse in the > opposite direction is less easy. > > The last time I fiddled a bit (yesterday as it happened) - it turned > out to be less than > blindingly obvious how to call other than trivial elixir code from erlang. > > I was also wondering about cross-compilation. Has anybody written > something that turns > erlang code into elixir source code or vice. versa. > > Cheers > > /Joe > > > > > > On Mon, Aug 7, 2017 at 3:46 PM, Andrew McIntyre > wrote: > > Hello Craig, > > > > Thanks for your help. > > > > I am trying to store the data as efficiently as possible. Its HL7 > > natively and this is my test: > > > > OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F > > > > |~^& are delimiters. The hierarchy is only so deep and using lists of > > lists to provide a tree like way to access the data eg Field 3, repeat > > 1 component 2 subcomponent1 > > > > Parsed it looks like this: > > > > [["OBX","17", > > ["FT","TEST"], > > [["8265-1",[],["LN","SUBCOMP"]]], > > [[["1","2","3","4"]]], > > "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] > > > > As the format evolves over time the hierarchy can be extended, but > > older clients can still read the value they are expecting if they > > follow the rules, like reading the first value in the list when you > > only expect one value to be there. > > > > Currently a typical system might have 12 million of these records so > > want to keep format as small as possible in the erlang format, hence > > reluctant to tag 2 much, but know how to get value of interest. Maybe > > that is my non erlang background showing up? Traversing 4 small lists > > by index should be fast?? > > > > I guess I could save strings as binary in the lists then is_binary > > should work?? Is that the case. I gather on 64bit system especially > > binary is more space efficient. > > > > Monday, August 7, 2017, 10:53:11 PM, you wrote: > > > > z> On 2017?08?07? ??? 22:29:31 you wrote: > >>> Hello zxq9, > >>> > >>> Thanks, Unfortunately I do not know the value of the string that will > >>> be there. Its an extensible hierarchy that can be several lists deep - > >>> or not. Might need to revise the data structure > > > > z> In this case it can be useful to consider a way of tagging values. > > > > z> Imagine we want to represent a directory tree structure and have a > > z> descent-first traversal function recurse over it while creating the > > z> tree. We have two things that can happen, there is a flat list of > > z> new directories that need to be created, and there is the > > z> possibility that the tree depth extends deeper at each node. > > > > z> The naive version would look like what you have: > > > > z> ["top_dir_1", > > z> "top_dir_2", > > z> ["next_level_1", > > z> "next_level_2"]] > > > > z> This leaves a bit to be desired, not only because of the problem > > z> you have pointed out that makes it difficult to know what is deep > > z> and what is shallow, but also because we don't really have a good > > z> way to represent a full tree (what would be the name of a directory > containing other directories?). > > > > z> So consider instead something like this: > > > > z> [{"top_dir_1", []}, > > z> {"top_dir_2", []}, > > z> {"top_dir_3", > > z> [{"next_level_1", []}, > > z> {"next_level_2", []}]}] > > > > z> Now we have a representation of each directory's name AND its > contents. > > > > z> We can traverse this laterally AND in depth without any ambiguity > > z> or need for carrying around a record of where we have been (by > > z> using depth recursion and tail-call recursion): > > > > > > z> make_tree([{Dir, Contents} | Rest]) -> > > z> ok = > > z> case filelib:is_dir(Dir) of > > z> true -> > > z> ok; > > z> false -> > > z> ok = log(info, "Creating dir: ~p", [Dir]), > > z> file:make_dir(Dir) > > z> end, > > z> ok = file:set_cwd(Dir), > > z> ok = make_tree(Contents), > > z> ok = file:set_cwd(".."), > > z> make_tree(Rest); > > make_tree([]) ->> > > z> ok. > > > > > > z> Not so bad. > > > > z> In your case we could represent things perhaps a bit better by > > z> separating the types and tagging them. Instead of just "FT" and > > z> whatever other string labels you might want, you could either use > > z> atoms (totally unambiguous) or tuples as we have in the example > > z> able (also totally unambiguous). I prefer tuples, though, because > they are easier to read. > > > > z> [{value, "foo"}, > > z> {tree, > > z> [{value, "bar"}, > > z> {value, "foo"}]}, > > z> {value, "baz"}] > > > > > > z> So then we do something like: > > > > > > z> traverse([{value, Value} | Rest]) -> > > z> ok = do_thing(Value), > > z> traverse(Rest); > > z> traverse([{tree, Contents} | Rest]) -> > > z> ok = traverse(Contents), > > z> traverse(Rest); > > traverse([]) ->> > > z> ok. > > > > > > z> Anyway, don't be afraid of varying your value types to say exactly > > z> what you mean. If your strings like "FT" only had meaning within > > z> your system consider NOT USING STRINGS, and using atoms instead. That > makes it even easier: > > > > > > z> [foo, > > z> bar, > > z> [foo, > > z> bar], > > z> foo] > > > > > > z> So then we can do: > > > > > > z> traverse([foo | Rest]) -> > > z> ok = do_foo(), > > z> traverse(Rest); > > z> traverse([bar | Rest]) -> > > z> ok = do_bar(), > > z> traverse(Rest); > > z> traverse([Value | Rest]) when is_list(Value) -> > > z> ok = traverse(Value), > > z> traverse(Rest); > > traverse([]) ->> > > z> ok. > > > > > > z> And of course, you can not use a guard if you want to match on a > > z> list shape in the listy clause there, but that is a minor detail. > > z> The point is to make your data types MEAN SOMETHING REASONABLE > > z> within your system. Use atoms when your values are meaningful only > > z> within your system. Strings are for the birds. > > > > z> -Craig > > z> _______________________________________________ > > z> erlang-questions mailing list > > z> erlang-questions@REDACTED > > z> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > -- > > Best regards, > > Andrew mailto:andrew@REDACTED > > > > sent from a real computer > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Best regards, Alex [Oleksii Semilietov] -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonas.trantina@REDACTED Tue Aug 8 11:13:25 2017 From: jonas.trantina@REDACTED (=?UTF-8?B?Sm9uw6HFoSBUcmFudGluYQ==?=) Date: Tue, 08 Aug 2017 09:13:25 +0000 Subject: [erlang-questions] List Question In-Reply-To: References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> Message-ID: Hi Joe, I recently investigated this topic and found the following rebar3 plugin you can use (if you are using rebar3) - https://github.com/barrel-db/rebar3_elixir_compile. I think it supports both your own Elixir code and Elixir dependencies, it automatically builds all of those so you can use them from your Erlang code. And using it from Erlang shouldn't be a problem once you know some Elixir basics (at least how the types are mapped between Ex and Erl). There's one plugin for old rebar too I believe. While I tried the above plugin and it seemed to work fine (on a smallish project) I found that using mix (Elixir's build tool) worked much better. Since mix supports both Erlang and Elixir out-of-the-box it seemed more natural for mixed projects (again tested only on a smallish test project, more info can be found at https://hexdocs.pm/mix/Mix.Tasks.Compile.Erlang.html#content). Regards, Jonas Trantina ?t 8. 8. 2017 v 10:01 odes?latel Joe Armstrong napsal: > Hello, > > I'm going to go way off topic here and not answer your specific > question about lists ... > > Your last mail had the information I need - you're trying to parse HL7. > I have a few comments. > > 1) Your original question did not bother to mention what problem you > were trying to solve - > You asked about a sub-problem that you encountered when trying to > solve your principle > problem (principle problem = parse HL7) (sub-problem = differentiate > lists) > > 2) It's *always* a good idea to ask questions about the principle > problem first !!!! > > I didn't know what HL7 was - my immediate thought was > 'I wonder if anybody has written an *proper* HL7 parser in Erlang' - by > proper I mean "has expended a significant amount of thought on writing a > parser" > > Google is your friend - It told me what HL7 was (I hadn't a clue here > - "never heard of it") > and it turned up a parser in elixir > > https://github.com/jcomellas/ex_hl7 > > From the quality of the documentation I assume this is a *proper* > implementation. > > Now elixir compiles to .beam files and can be called from Erlang - > which raises another > sub problem "how do I compile the elixir code and call it from Erlang" > and begs the > question "is this effort worthwhile" > > Given that a parser for HL7 exists in elixir it might be sensible to > use it "off the shelf" > > I have a feeling that elixir folks are good at reusing erlang code - > but that reuse in the > opposite direction is less easy. > > The last time I fiddled a bit (yesterday as it happened) - it turned > out to be less than > blindingly obvious how to call other than trivial elixir code from erlang. > > I was also wondering about cross-compilation. Has anybody written > something that turns > erlang code into elixir source code or vice. versa. > > Cheers > > /Joe > > > > > > On Mon, Aug 7, 2017 at 3:46 PM, Andrew McIntyre > wrote: > > Hello Craig, > > > > Thanks for your help. > > > > I am trying to store the data as efficiently as possible. Its HL7 > > natively and this is my test: > > > > OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F > > > > |~^& are delimiters. The hierarchy is only so deep and using lists of > > lists to provide a tree like way to access the data eg Field 3, repeat > > 1 component 2 subcomponent1 > > > > Parsed it looks like this: > > > > [["OBX","17", > > ["FT","TEST"], > > [["8265-1",[],["LN","SUBCOMP"]]], > > [[["1","2","3","4"]]], > > "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] > > > > As the format evolves over time the hierarchy can be extended, but > > older clients can still read the value they are expecting if they > > follow the rules, like reading the first value in the list when you > > only expect one value to be there. > > > > Currently a typical system might have 12 million of these records so > > want to keep format as small as possible in the erlang format, hence > > reluctant to tag 2 much, but know how to get value of interest. Maybe > > that is my non erlang background showing up? Traversing 4 small lists > > by index should be fast?? > > > > I guess I could save strings as binary in the lists then is_binary > > should work?? Is that the case. I gather on 64bit system especially > > binary is more space efficient. > > > > Monday, August 7, 2017, 10:53:11 PM, you wrote: > > > > z> On 2017?08?07? ??? 22:29:31 you wrote: > >>> Hello zxq9, > >>> > >>> Thanks, Unfortunately I do not know the value of the string that will > >>> be there. Its an extensible hierarchy that can be several lists deep - > >>> or not. Might need to revise the data structure > > > > z> In this case it can be useful to consider a way of tagging values. > > > > z> Imagine we want to represent a directory tree structure and have a > > z> descent-first traversal function recurse over it while creating the > > z> tree. We have two things that can happen, there is a flat list of > > z> new directories that need to be created, and there is the > > z> possibility that the tree depth extends deeper at each node. > > > > z> The naive version would look like what you have: > > > > z> ["top_dir_1", > > z> "top_dir_2", > > z> ["next_level_1", > > z> "next_level_2"]] > > > > z> This leaves a bit to be desired, not only because of the problem > > z> you have pointed out that makes it difficult to know what is deep > > z> and what is shallow, but also because we don't really have a good > > z> way to represent a full tree (what would be the name of a directory > containing other directories?). > > > > z> So consider instead something like this: > > > > z> [{"top_dir_1", []}, > > z> {"top_dir_2", []}, > > z> {"top_dir_3", > > z> [{"next_level_1", []}, > > z> {"next_level_2", []}]}] > > > > z> Now we have a representation of each directory's name AND its > contents. > > > > z> We can traverse this laterally AND in depth without any ambiguity > > z> or need for carrying around a record of where we have been (by > > z> using depth recursion and tail-call recursion): > > > > > > z> make_tree([{Dir, Contents} | Rest]) -> > > z> ok = > > z> case filelib:is_dir(Dir) of > > z> true -> > > z> ok; > > z> false -> > > z> ok = log(info, "Creating dir: ~p", [Dir]), > > z> file:make_dir(Dir) > > z> end, > > z> ok = file:set_cwd(Dir), > > z> ok = make_tree(Contents), > > z> ok = file:set_cwd(".."), > > z> make_tree(Rest); > > make_tree([]) ->> > > z> ok. > > > > > > z> Not so bad. > > > > z> In your case we could represent things perhaps a bit better by > > z> separating the types and tagging them. Instead of just "FT" and > > z> whatever other string labels you might want, you could either use > > z> atoms (totally unambiguous) or tuples as we have in the example > > z> able (also totally unambiguous). I prefer tuples, though, because > they are easier to read. > > > > z> [{value, "foo"}, > > z> {tree, > > z> [{value, "bar"}, > > z> {value, "foo"}]}, > > z> {value, "baz"}] > > > > > > z> So then we do something like: > > > > > > z> traverse([{value, Value} | Rest]) -> > > z> ok = do_thing(Value), > > z> traverse(Rest); > > z> traverse([{tree, Contents} | Rest]) -> > > z> ok = traverse(Contents), > > z> traverse(Rest); > > traverse([]) ->> > > z> ok. > > > > > > z> Anyway, don't be afraid of varying your value types to say exactly > > z> what you mean. If your strings like "FT" only had meaning within > > z> your system consider NOT USING STRINGS, and using atoms instead. That > makes it even easier: > > > > > > z> [foo, > > z> bar, > > z> [foo, > > z> bar], > > z> foo] > > > > > > z> So then we can do: > > > > > > z> traverse([foo | Rest]) -> > > z> ok = do_foo(), > > z> traverse(Rest); > > z> traverse([bar | Rest]) -> > > z> ok = do_bar(), > > z> traverse(Rest); > > z> traverse([Value | Rest]) when is_list(Value) -> > > z> ok = traverse(Value), > > z> traverse(Rest); > > traverse([]) ->> > > z> ok. > > > > > > z> And of course, you can not use a guard if you want to match on a > > z> list shape in the listy clause there, but that is a minor detail. > > z> The point is to make your data types MEAN SOMETHING REASONABLE > > z> within your system. Use atoms when your values are meaningful only > > z> within your system. Strings are for the birds. > > > > z> -Craig > > z> _______________________________________________ > > z> erlang-questions mailing list > > z> erlang-questions@REDACTED > > z> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > -- > > Best regards, > > Andrew mailto:andrew@REDACTED > > > > sent from a real computer > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Tue Aug 8 11:25:09 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 8 Aug 2017 12:25:09 +0300 Subject: [erlang-questions] Memory leak in SSL In-Reply-To: References: Message-ID: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> Hello, Have you tried to disable reuse of ssl sessions? {ssl_options, [{reuse_sessions, false}]} And reduce the session time -ssl session_lifetime 120 Best Regards, Dmitry > On 8 Aug 2017, at 11.59, Max Treskin wrote: > > Hello, > > I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks. > ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM. > What should I do to eliminate this? Is it bug or intended behaviour? > > Thanks > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From andrew@REDACTED Tue Aug 8 11:52:52 2017 From: andrew@REDACTED (Andrew McIntyre) Date: Tue, 8 Aug 2017 19:52:52 +1000 Subject: [erlang-questions] List Question In-Reply-To: References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> Message-ID: <45821730.20170808195252@Medical-Objects.com.au> Hello Joe, Thanks for you thoughts, I have been following erlang since your first book, but have 20yrs of OO Delphi code that parses hL7 and I was trying to learn erlang by implementing something I am very familiar with in an OO way, in a functional way, I am sure will take a while to get a hang of the community and functional programming. Have a bad habit of trying to implement everything myself, and HL7 has nuances that make backward and forward compatibility quite good and that starts in parser. We have also done in elixir, but a little to much magic and syntax inconsistencies for my taste in elixir. I guess thats because I am used to very readable explicit pascal code, but elixir appeals to people coming from ruby. In the end it runs on the beam, and that is what attracts me! Will try and explain questions better, thanks for all the responses Andrew Tuesday, August 8, 2017, 6:00:59 PM, you wrote: JA> Hello, JA> I'm going to go way off topic here and not answer your specific JA> question about lists ... JA> Your last mail had the information I need - you're trying to parse HL7. JA> I have a few comments. JA> 1) Your original question did not bother to mention what problem you JA> were trying to solve - JA> You asked about a sub-problem that you encountered when trying to JA> solve your principle JA> problem (principle problem = parse HL7) (sub-problem = differentiate lists) JA> 2) It's *always* a good idea to ask questions about the principle JA> problem first !!!! JA> I didn't know what HL7 was - my immediate thought was JA> 'I wonder if anybody has written an *proper* HL7 parser in Erlang' - by JA> proper I mean "has expended a significant amount of thought on writing a parser" JA> Google is your friend - It told me what HL7 was (I hadn't a clue here JA> - "never heard of it") JA> and it turned up a parser in elixir JA> https://github.com/jcomellas/ex_hl7 >>From the quality of the documentation I assume this is a *proper* JA> implementation. JA> Now elixir compiles to .beam files and can be called from Erlang - JA> which raises another JA> sub problem "how do I compile the elixir code and call it from Erlang" JA> and begs the JA> question "is this effort worthwhile" JA> Given that a parser for HL7 exists in elixir it might be sensible to JA> use it "off the shelf" JA> I have a feeling that elixir folks are good at reusing erlang code - JA> but that reuse in the JA> opposite direction is less easy. JA> The last time I fiddled a bit (yesterday as it happened) - it turned JA> out to be less than JA> blindingly obvious how to call other than trivial elixir code from erlang. JA> I was also wondering about cross-compilation. Has anybody written JA> something that turns JA> erlang code into elixir source code or vice. versa. JA> Cheers JA> /Joe JA> On Mon, Aug 7, 2017 at 3:46 PM, Andrew McIntyre JA> wrote: >> Hello Craig, >> >> Thanks for your help. >> >> I am trying to store the data as efficiently as possible. Its HL7 >> natively and this is my test: >> >> OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F >> >> |~^& are delimiters. The hierarchy is only so deep and using lists of >> lists to provide a tree like way to access the data eg Field 3, repeat >> 1 component 2 subcomponent1 >> >> Parsed it looks like this: >> >> [["OBX","17", >> ["FT","TEST"], >> [["8265-1",[],["LN","SUBCOMP"]]], >> [[["1","2","3","4"]]], >> "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] >> >> As the format evolves over time the hierarchy can be extended, but >> older clients can still read the value they are expecting if they >> follow the rules, like reading the first value in the list when you >> only expect one value to be there. >> >> Currently a typical system might have 12 million of these records so >> want to keep format as small as possible in the erlang format, hence >> reluctant to tag 2 much, but know how to get value of interest. Maybe >> that is my non erlang background showing up? Traversing 4 small lists >> by index should be fast?? >> >> I guess I could save strings as binary in the lists then is_binary >> should work?? Is that the case. I gather on 64bit system especially >> binary is more space efficient. >> >> Monday, August 7, 2017, 10:53:11 PM, you wrote: >> >> z> On 2017?08?07? ??? 22:29:31 you wrote: >>>> Hello zxq9, >>>> >>>> Thanks, Unfortunately I do not know the value of the string that will >>>> be there. Its an extensible hierarchy that can be several lists deep - >>>> or not. Might need to revise the data structure >> >> z> In this case it can be useful to consider a way of tagging values. >> >> z> Imagine we want to represent a directory tree structure and have a >> z> descent-first traversal function recurse over it while creating the >> z> tree. We have two things that can happen, there is a flat list of >> z> new directories that need to be created, and there is the >> z> possibility that the tree depth extends deeper at each node. >> >> z> The naive version would look like what you have: >> >> z> ["top_dir_1", >> z> "top_dir_2", >> z> ["next_level_1", >> z> "next_level_2"]] >> >> z> This leaves a bit to be desired, not only because of the problem >> z> you have pointed out that makes it difficult to know what is deep >> z> and what is shallow, but also because we don't really have a good >> z> way to represent a full tree (what would be the name of a directory containing other directories?). >> >> z> So consider instead something like this: >> >> z> [{"top_dir_1", []}, >> z> {"top_dir_2", []}, >> z> {"top_dir_3", >> z> [{"next_level_1", []}, >> z> {"next_level_2", []}]}] >> >> z> Now we have a representation of each directory's name AND its contents. >> >> z> We can traverse this laterally AND in depth without any ambiguity >> z> or need for carrying around a record of where we have been (by >> z> using depth recursion and tail-call recursion): >> >> >> z> make_tree([{Dir, Contents} | Rest]) -> >> z> ok = >> z> case filelib:is_dir(Dir) of >> z> true -> >> z> ok; >> z> false -> >> z> ok = log(info, "Creating dir: ~p", [Dir]), >> z> file:make_dir(Dir) >> z> end, >> z> ok = file:set_cwd(Dir), >> z> ok = make_tree(Contents), >> z> ok = file:set_cwd(".."), >> z> make_tree(Rest); >> make_tree([]) ->> >> z> ok. >> >> >> z> Not so bad. >> >> z> In your case we could represent things perhaps a bit better by >> z> separating the types and tagging them. Instead of just "FT" and >> z> whatever other string labels you might want, you could either use >> z> atoms (totally unambiguous) or tuples as we have in the example >> z> able (also totally unambiguous). I prefer tuples, though, because they are easier to read. >> >> z> [{value, "foo"}, >> z> {tree, >> z> [{value, "bar"}, >> z> {value, "foo"}]}, >> z> {value, "baz"}] >> >> >> z> So then we do something like: >> >> >> z> traverse([{value, Value} | Rest]) -> >> z> ok = do_thing(Value), >> z> traverse(Rest); >> z> traverse([{tree, Contents} | Rest]) -> >> z> ok = traverse(Contents), >> z> traverse(Rest); >> traverse([]) ->> >> z> ok. >> >> >> z> Anyway, don't be afraid of varying your value types to say exactly >> z> what you mean. If your strings like "FT" only had meaning within >> z> your system consider NOT USING STRINGS, and using atoms instead. That makes it even easier: >> >> >> z> [foo, >> z> bar, >> z> [foo, >> z> bar], >> z> foo] >> >> >> z> So then we can do: >> >> >> z> traverse([foo | Rest]) -> >> z> ok = do_foo(), >> z> traverse(Rest); >> z> traverse([bar | Rest]) -> >> z> ok = do_bar(), >> z> traverse(Rest); >> z> traverse([Value | Rest]) when is_list(Value) -> >> z> ok = traverse(Value), >> z> traverse(Rest); >> traverse([]) ->> >> z> ok. >> >> >> z> And of course, you can not use a guard if you want to match on a >> z> list shape in the listy clause there, but that is a minor detail. >> z> The point is to make your data types MEAN SOMETHING REASONABLE >> z> within your system. Use atoms when your values are meaningful only >> z> within your system. Strings are for the birds. >> >> z> -Craig >> z> _______________________________________________ >> z> erlang-questions mailing list >> z> erlang-questions@REDACTED >> z> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> -- >> Best regards, >> Andrew mailto:andrew@REDACTED >> >> sent from a real computer >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Andrew mailto:andrew@REDACTED sent from a real computer From erlang@REDACTED Tue Aug 8 12:23:00 2017 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 8 Aug 2017 12:23:00 +0200 Subject: [erlang-questions] List Question In-Reply-To: <45821730.20170808195252@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> <45821730.20170808195252@Medical-Objects.com.au> Message-ID: On Tue, Aug 8, 2017 at 11:52 AM, Andrew McIntyre wrote: > Hello Joe, > > Thanks for you thoughts, I have been following erlang since your first > book, but have 20yrs of OO Delphi code that parses hL7 and I was trying to > learn erlang by implementing something I am very familiar with in an > OO way, in a functional way, I am sure will take a while to get a hang > of the community and functional programming. > > Have a bad habit of > trying to implement everything myself, I have this too - I think it's a very good habit. There are only a few ways to *really* learn things - Implement it yourself, teach it, write a book about it. /Joe > and HL7 has nuances that make > backward and forward compatibility quite good and that starts in > parser. We have also done in elixir, but a little to much magic and > syntax inconsistencies for my taste in elixir. I guess thats because I > am used to very readable explicit pascal code, but elixir appeals to > people coming from ruby. In the end it runs on the beam, and that is > what attracts me! > > Will try and explain questions better, thanks for all the responses > > Andrew > > > Tuesday, August 8, 2017, 6:00:59 PM, you wrote: > > JA> Hello, > > JA> I'm going to go way off topic here and not answer your specific > JA> question about lists ... > > JA> Your last mail had the information I need - you're trying to parse HL7. > JA> I have a few comments. > > JA> 1) Your original question did not bother to mention what problem you > JA> were trying to solve - > JA> You asked about a sub-problem that you encountered when trying to > JA> solve your principle > JA> problem (principle problem = parse HL7) (sub-problem = differentiate lists) > > JA> 2) It's *always* a good idea to ask questions about the principle > JA> problem first !!!! > > JA> I didn't know what HL7 was - my immediate thought was > JA> 'I wonder if anybody has written an *proper* HL7 parser in Erlang' - by > JA> proper I mean "has expended a significant amount of thought on writing a parser" > > JA> Google is your friend - It told me what HL7 was (I hadn't a clue here > JA> - "never heard of it") > JA> and it turned up a parser in elixir > > JA> https://github.com/jcomellas/ex_hl7 > >>>From the quality of the documentation I assume this is a *proper* > JA> implementation. > > JA> Now elixir compiles to .beam files and can be called from Erlang - > JA> which raises another > JA> sub problem "how do I compile the elixir code and call it from Erlang" > JA> and begs the > JA> question "is this effort worthwhile" > > JA> Given that a parser for HL7 exists in elixir it might be sensible to > JA> use it "off the shelf" > > JA> I have a feeling that elixir folks are good at reusing erlang code - > JA> but that reuse in the > JA> opposite direction is less easy. > > JA> The last time I fiddled a bit (yesterday as it happened) - it turned > JA> out to be less than > JA> blindingly obvious how to call other than trivial elixir code from erlang. > > JA> I was also wondering about cross-compilation. Has anybody written > JA> something that turns > JA> erlang code into elixir source code or vice. versa. > > JA> Cheers > > JA> /Joe > > > > > > JA> On Mon, Aug 7, 2017 at 3:46 PM, Andrew McIntyre > JA> wrote: >>> Hello Craig, >>> >>> Thanks for your help. >>> >>> I am trying to store the data as efficiently as possible. Its HL7 >>> natively and this is my test: >>> >>> OBX|17|FT~TEST|8265-1^^LN&SUBCOMP|1&2&3&4|\H\Spot Image 2\N\||||||F >>> >>> |~^& are delimiters. The hierarchy is only so deep and using lists of >>> lists to provide a tree like way to access the data eg Field 3, repeat >>> 1 component 2 subcomponent1 >>> >>> Parsed it looks like this: >>> >>> [["OBX","17", >>> ["FT","TEST"], >>> [["8265-1",[],["LN","SUBCOMP"]]], >>> [[["1","2","3","4"]]], >>> "\\H\\Spot Image 2\\N\\",[],[],[],[],[],"F"]] >>> >>> As the format evolves over time the hierarchy can be extended, but >>> older clients can still read the value they are expecting if they >>> follow the rules, like reading the first value in the list when you >>> only expect one value to be there. >>> >>> Currently a typical system might have 12 million of these records so >>> want to keep format as small as possible in the erlang format, hence >>> reluctant to tag 2 much, but know how to get value of interest. Maybe >>> that is my non erlang background showing up? Traversing 4 small lists >>> by index should be fast?? >>> >>> I guess I could save strings as binary in the lists then is_binary >>> should work?? Is that the case. I gather on 64bit system especially >>> binary is more space efficient. >>> >>> Monday, August 7, 2017, 10:53:11 PM, you wrote: >>> >>> z> On 2017?08?07? ??? 22:29:31 you wrote: >>>>> Hello zxq9, >>>>> >>>>> Thanks, Unfortunately I do not know the value of the string that will >>>>> be there. Its an extensible hierarchy that can be several lists deep - >>>>> or not. Might need to revise the data structure >>> >>> z> In this case it can be useful to consider a way of tagging values. >>> >>> z> Imagine we want to represent a directory tree structure and have a >>> z> descent-first traversal function recurse over it while creating the >>> z> tree. We have two things that can happen, there is a flat list of >>> z> new directories that need to be created, and there is the >>> z> possibility that the tree depth extends deeper at each node. >>> >>> z> The naive version would look like what you have: >>> >>> z> ["top_dir_1", >>> z> "top_dir_2", >>> z> ["next_level_1", >>> z> "next_level_2"]] >>> >>> z> This leaves a bit to be desired, not only because of the problem >>> z> you have pointed out that makes it difficult to know what is deep >>> z> and what is shallow, but also because we don't really have a good >>> z> way to represent a full tree (what would be the name of a directory containing other directories?). >>> >>> z> So consider instead something like this: >>> >>> z> [{"top_dir_1", []}, >>> z> {"top_dir_2", []}, >>> z> {"top_dir_3", >>> z> [{"next_level_1", []}, >>> z> {"next_level_2", []}]}] >>> >>> z> Now we have a representation of each directory's name AND its contents. >>> >>> z> We can traverse this laterally AND in depth without any ambiguity >>> z> or need for carrying around a record of where we have been (by >>> z> using depth recursion and tail-call recursion): >>> >>> >>> z> make_tree([{Dir, Contents} | Rest]) -> >>> z> ok = >>> z> case filelib:is_dir(Dir) of >>> z> true -> >>> z> ok; >>> z> false -> >>> z> ok = log(info, "Creating dir: ~p", [Dir]), >>> z> file:make_dir(Dir) >>> z> end, >>> z> ok = file:set_cwd(Dir), >>> z> ok = make_tree(Contents), >>> z> ok = file:set_cwd(".."), >>> z> make_tree(Rest); >>> make_tree([]) ->> >>> z> ok. >>> >>> >>> z> Not so bad. >>> >>> z> In your case we could represent things perhaps a bit better by >>> z> separating the types and tagging them. Instead of just "FT" and >>> z> whatever other string labels you might want, you could either use >>> z> atoms (totally unambiguous) or tuples as we have in the example >>> z> able (also totally unambiguous). I prefer tuples, though, because they are easier to read. >>> >>> z> [{value, "foo"}, >>> z> {tree, >>> z> [{value, "bar"}, >>> z> {value, "foo"}]}, >>> z> {value, "baz"}] >>> >>> >>> z> So then we do something like: >>> >>> >>> z> traverse([{value, Value} | Rest]) -> >>> z> ok = do_thing(Value), >>> z> traverse(Rest); >>> z> traverse([{tree, Contents} | Rest]) -> >>> z> ok = traverse(Contents), >>> z> traverse(Rest); >>> traverse([]) ->> >>> z> ok. >>> >>> >>> z> Anyway, don't be afraid of varying your value types to say exactly >>> z> what you mean. If your strings like "FT" only had meaning within >>> z> your system consider NOT USING STRINGS, and using atoms instead. That makes it even easier: >>> >>> >>> z> [foo, >>> z> bar, >>> z> [foo, >>> z> bar], >>> z> foo] >>> >>> >>> z> So then we can do: >>> >>> >>> z> traverse([foo | Rest]) -> >>> z> ok = do_foo(), >>> z> traverse(Rest); >>> z> traverse([bar | Rest]) -> >>> z> ok = do_bar(), >>> z> traverse(Rest); >>> z> traverse([Value | Rest]) when is_list(Value) -> >>> z> ok = traverse(Value), >>> z> traverse(Rest); >>> traverse([]) ->> >>> z> ok. >>> >>> >>> z> And of course, you can not use a guard if you want to match on a >>> z> list shape in the listy clause there, but that is a minor detail. >>> z> The point is to make your data types MEAN SOMETHING REASONABLE >>> z> within your system. Use atoms when your values are meaningful only >>> z> within your system. Strings are for the birds. >>> >>> z> -Craig >>> z> _______________________________________________ >>> z> erlang-questions mailing list >>> z> erlang-questions@REDACTED >>> z> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >>> -- >>> Best regards, >>> Andrew mailto:andrew@REDACTED >>> >>> sent from a real computer >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions > > > > > -- > Best regards, > Andrew mailto:andrew@REDACTED > > sent from a real computer > > From erlang@REDACTED Tue Aug 8 15:36:02 2017 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 8 Aug 2017 15:36:02 +0200 Subject: [erlang-questions] List Question In-Reply-To: <59897AE3.6040609@gmail.com> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> <59897AE3.6040609@gmail.com> Message-ID: Thanks - great fun. Makes for an easy way to learn Elixir. Just transpile into Erlang and read the code :-) /Joe On Tue, Aug 8, 2017 at 10:48 AM, Michael Truog wrote: > On 08/08/2017 01:00 AM, Joe Armstrong wrote: >> >> The last time I fiddled a bit (yesterday as it happened) - it turned >> out to be less than >> blindingly obvious how to call other than trivial elixir code from erlang. >> >> I was also wondering about cross-compilation. Has anybody written >> something that turns >> erlang code into elixir source code or vice. versa. > > > I created a script ex2erl > (https://github.com/okeuday/reltool_util/blob/master/ex2erl) to convert a > single Elixir module into multiple Erlang modules. Then it is easy to > include Elixir source code into an Erlang project without creating a > dependency on Elixir for all use of the Erlang project. The ex2erl script > requires an Elixir installation alongside the Erlang installation and > depends on the Elixir module having the debug_info in the beam output which > is done by default. > > Best Regards, > Michael From ingela.andin@REDACTED Tue Aug 8 15:55:21 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Tue, 8 Aug 2017 15:55:21 +0200 Subject: [erlang-questions] Memory leak in SSL In-Reply-To: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> References: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> Message-ID: Hi! Depending on the version of the ssl application you may also set these values. I guess you have an older version as they default to 1000. >From doc: *"session_cache_client_max = integer() * Limits the growth of the clients session cache, that is how many sessions towards servers that are cached to be used by new client connections. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000. * session_cache_server_max = integer() * Limits the growth of the servers session cache, that is how many client sessions are cached by the server. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000." Regards Ingela Erlang/OTP Team - Ericsson AB 2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov : > Hello, > > Have you tried to disable reuse of ssl sessions? > {ssl_options, [{reuse_sessions, false}]} > > And reduce the session time > -ssl session_lifetime 120 > > Best Regards, > Dmitry > > > On 8 Aug 2017, at 11.59, Max Treskin wrote: > > > > Hello, > > > > I have two different HTTPS-servers (different purposes, code, etc) built > on top of cowboy/ranch, and both have memory leaks. > > ETS table server_ssl_otp_session_cache has millions of records just > after hours of work and consumes gigabytes of RAM. > > What should I do to eliminate this? Is it bug or intended behaviour? > > > > Thanks > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zerthurd@REDACTED Tue Aug 8 15:54:49 2017 From: zerthurd@REDACTED (Max Treskin) Date: Tue, 8 Aug 2017 16:54:49 +0300 Subject: [erlang-questions] Memory leak in SSL In-Reply-To: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> References: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> Message-ID: Thank you Dmitry! It works On 8 August 2017 at 12:25:12, Dmitry Kolesnikov (dmkolesnikov@REDACTED) wrote: Hello, Have you tried to disable reuse of ssl sessions? {ssl_options, [{reuse_sessions, false}]} And reduce the session time -ssl session_lifetime 120 Best Regards, Dmitry > On 8 Aug 2017, at 11.59, Max Treskin wrote: > > Hello, > > I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks. > ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM. > What should I do to eliminate this? Is it bug or intended behaviour? > > Thanks > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Tue Aug 8 20:10:33 2017 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Tue, 8 Aug 2017 11:10:33 -0700 Subject: [erlang-questions] Erlang 20 and crypto:rand_uniform/2 In-Reply-To: <20170517075122.GA68388@erix.ericsson.se> References: <20170517001632.841B3994@m0087794.ppops.net> <20170517075122.GA68388@erix.ericsson.se> Message-ID: I am sorry but nothing of what you said is even close to a good reason to not provide a /2 providing the same API as crypto:rand_uniform/2. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raugerfacebook@REDACTED Tue Aug 8 21:18:27 2017 From: raugerfacebook@REDACTED (Ryan Auger) Date: Tue, 8 Aug 2017 15:18:27 -0400 Subject: [erlang-questions] Retrieving TLS Information After/On Connect Message-ID: <1D29362A-490C-4A6F-ADD2-06C9FF6B4E6E@gmail.com> Hello, I am using PSKs to connect to my VMQTT/EMQTT instance, and need to modify the source code or write a plugin to set my clientid or username = psk-identity, thus creating a ?TLS-PSK based authentication?. To be clear, my connection using PSKs works, and I can connect and send encrypted messages using an incredibly basic user_lookup_fun like so: user_lookup(psk, _Username, UserState) -> {ok, UserState}. The problem here is that I have no reference to the calling process, and can?t modify the ClientID or username, so I cannot authenticate users without sending an additional username and password. If someone gained access to my private key, they would be able to publish and subscribe to any topic because the psk_identity is not currently used in authentication. The way I see it, there are two options: 1) Set the ClientID in user lookup -> this is impossible because I have no reference to the calling process 2) Find where the ClientID is set, use a function to get the psk-identity from the listening process, and then set the ClientID = psk-identity For option 2, my first goal is to just get the psk_identity from somewhere in the code. I tried to use the erlang library function ssl_connection:connection_information(Client#mqtt_client.client_pid)]) on the client.connected hook, But this process hangs and does not return any value. Can anyone point me in the right direction for getting the TLS information from within the code? Otherwise, is there any way that I could go about option 1) without knowing anything about the calling process? Thank you in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstimpson@REDACTED Tue Aug 8 23:24:54 2017 From: jstimpson@REDACTED (Jesse Stimpson) Date: Tue, 8 Aug 2017 17:24:54 -0400 Subject: [erlang-questions] Slave node crashes when using proto_dist inet_tls Message-ID: Is it possible to start a slave node from a master that uses SSL for Erlang distribution? I have a valid bootfile, and I've been able to get regular rpc between two distinct nodes working, However, when I use slave:start/3 with all the necessary args, my slave crashes with the error below. slave:start_link("localhost", jms, "-boot start_ssl -proto_dist inet_tls -ssl_dist_opt server_certfile /path/to/server.pem -ssl_dist_opt server_verify verify_none -ssl_dist_opt server_depth 0 -ssl_dist_opt server_fail_if_no_peer_cert false -ssl_dist_opt client_certfile /path/to/client.pem -ssl_dist_opt client_verify verify_none -ssl_dist_opt client_depth 0 -setcookie mycookie -env ERL_CRASH_DUMP erl_crash_dump.slave"). In erl_crash_dump.slave: Slogan: Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,file_server_2,{no_master,{badrpc,nodedown}}}},{kernel,start,[normal,[]]}}}) A cursory reading of the code shows that during the kernel startup, the file_server is attempting an rpc, and failing, presumably because the SSL stack has not yet been started. Is there any way around this? Thanks, Jesse -- Jesse Stimpson Site Reliability Engineering m: 9199950424 <(919)%20995-0424> RepublicWireless.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Aug 9 01:27:24 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 9 Aug 2017 11:27:24 +1200 Subject: [erlang-questions] List Question In-Reply-To: <45821730.20170808195252@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> <45821730.20170808195252@Medical-Objects.com.au> Message-ID: From what I've been able to glean about the HL7 message format, there are two aspects: * the basic syntax is the multi-level delimited thingy but * there is also *semantics*, predefined message types, an assortment of data types, rules for mapping data types to trees and so on. From a quick look at the Elixir HL7 parser, they have taken steps to handle some (but perhaps not all) of the *semantics* of HL7 and don't just give a tree of strings, but more structured data. Parsing the multi-level delimited syntax is trivial. Dealing with the semantics is not. I think that in figuring out for yourself how to work with HL7 messages in Erlang, the starting point would be - what message types do you want to handle? - what kinds of data occur in them? - how do you want to represent those kinds of data in Erlang? - do you actually want to represent *every* field at all? Some might not be relevant to you. - would you be streaming messages through a system (like some sort of pub/sub queueing middleware), summarising messages, storing them, or what? - what does a type declaration for a message type look like in HL7? Is there some way to automatically derive parsing code from that? What I'm getting at with the last point is that there is ASN.1 support for Erlang. Give it an ASN.1 definition, and you get Erlang code out the other end. I am particularly thinking of the PADS project: PADS: Processing Arbitrary Data Streams Kathleen Fisher and Bob Gruber, AT&T Labs Slides in ppt http://homepages.inf.ed.ac.uk/wadler/xmlbinding/ Transactional data streams, such as sequences of stock-market buy/sell orders, credit-card purchase records, web server entries, and electronic fund transfer orders, can be mined very profitably. As an example, researchers at AT&T have built customer profiles from streams of call-detail records to significant financial effect. Often such streams are high-volume: AT&T's call-detail stream contains roughly 300 million calls per day requiring approximately 7GBs of storage space. Typically, such stream data arrives ``as is'' in ad hoc formats with poor documentation. In addition, the data frequently contains errors. The appropriate response to such errors is application-specific. Some applications can simply discard unexpected or erroneous values and continue processing. For other applications, however, errors in the data can be the most interesting part of the data. Understanding a new data source and producing a suitable parser are crucial first steps in any use of such data. Unfortunately, writing parsers for this kind of data is a difficult task, both tedious and error-prone. It is complicated by lack of documentation, convoluted encodings designed to save space, the need to handle errors robustly, and the need to produce efficient code to cope with the scale of the stream. Often, the hard-won understanding of the data ends up embedded in parsing code, making long-term maintenance difficult for the original writer and sharing the knowledge with others nearly impossible. The goal of the PADS project is to provide languages and tools for simplifying data processing. We have a preliminary design of a declarative data-description language, PADSL, expressive enough to describe the data feeds we see at AT&T in practice, including ASCII, binary, EBCDIC, Cobol, and mixed data formats. From PADSL we generate a tunable C library with functions for parsing, manipulating, and summarizing the data. In joint work with Mary Fernandez and Ricardo Medel, we are working to integrate PADS and XQuery to support declarative querying of data sources with PADS descriptions. -------------------- The PADS project moved from AT&T to http://pads.cs.tufts.edu/doc.html I say this in all seriousness: if I had a need to process GB of HL7 data, I would start by seeing if PADS was adequate to describe it, and if so I'd write an HL7->Erlang data translator in C or ML (as PADS has C and ML versions). If not, I'd see what ideas I could steal from PADS. Using a declarative data language to describe the message types I was interested in would be an up-front cost, but it would hugely simplify later maintenance. From ok@REDACTED Wed Aug 9 01:33:38 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 9 Aug 2017 11:33:38 +1200 Subject: [erlang-questions] List Question In-Reply-To: References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> <45821730.20170808195252@Medical-Objects.com.au> Message-ID: PS: one amazing thing with PADS is LearnPADS. You give it a bunch of data and it tries to figure out the format for itself. It would be very interesting to see what LearnPADS made of a bunch of HL7 data. From andrew@REDACTED Wed Aug 9 01:40:22 2017 From: andrew@REDACTED (Andrew McIntyre) Date: Wed, 9 Aug 2017 09:40:22 +1000 Subject: [erlang-questions] List Question In-Reply-To: References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> <45821730.20170808195252@Medical-Objects.com.au> Message-ID: <1523137591.20170809094022@Medical-Objects.com.au> Hello Richard, The oo implementation we have is in 2 parts, 1. parse into a tree with no knowledge of semantics 2. Use a specific Object to provide an interface to the data based on its type I am involved in standards process and there is a blog post here for anyone interested: https://kb.medical-objects.com.au/display/PUB/HL7v2+parsing Essestially plan to do same in erlang 1. Parse into a tree 2. Create unit that has functions for reading the known values for a specific segment - these are autogenerated eg have a msh.erl that has a msh:Sending_Facility() function 3. Use reader functions that allow message to become more complex, but permit old function to continue to work Andrew Wednesday, August 9, 2017, 9:27:24 AM, you wrote: RAOK> From what I've been able to glean about the HL7 message RAOK> format, there are two aspects: RAOK> * the basic syntax is the multi-level delimited thingy but RAOK> * there is also *semantics*, predefined message types, RAOK> an assortment of data types, rules for mapping data RAOK> types to trees and so on. RAOK> From a quick look at the Elixir HL7 parser, they have RAOK> taken steps to handle some (but perhaps not all) of the RAOK> *semantics* of HL7 and don't just give a tree of strings, RAOK> but more structured data. RAOK> Parsing the multi-level delimited syntax is trivial. RAOK> Dealing with the semantics is not. RAOK> I think that in figuring out for yourself how to work RAOK> with HL7 messages in Erlang, the starting point would RAOK> be RAOK> - what message types do you want to handle? RAOK> - what kinds of data occur in them? RAOK> - how do you want to represent those kinds of RAOK> data in Erlang? RAOK> - do you actually want to represent *every* field RAOK> at all? Some might not be relevant to you. RAOK> - would you be streaming messages through a RAOK> system (like some sort of pub/sub queueing RAOK> middleware), summarising messages, storing RAOK> them, or what? RAOK> - what does a type declaration for a message type RAOK> look like in HL7? Is there some way to automatically RAOK> derive parsing code from that? RAOK> What I'm getting at with the last point is that there RAOK> is ASN.1 support for Erlang. Give it an ASN.1 RAOK> definition, and you get Erlang code out the other end. RAOK> I am particularly thinking of the PADS project: RAOK> PADS: Processing Arbitrary Data Streams RAOK> Kathleen Fisher and Bob Gruber, AT&T Labs RAOK> Slides in ppt http://homepages.inf.ed.ac.uk/wadler/xmlbinding/ RAOK> Transactional data streams, such as sequences of stock-market RAOK> buy/sell orders, credit-card purchase records, web server RAOK> entries, and electronic fund transfer orders, can be mined very RAOK> profitably. As an example, researchers at AT&T have built RAOK> customer profiles from streams of call-detail records to significant financial effect. RAOK> Often such streams are high-volume: AT&T's call-detail stream RAOK> contains roughly 300 million calls per day requiring RAOK> approximately 7GBs of storage space. Typically, such stream data RAOK> arrives ``as is'' in ad hoc formats with poor documentation. In RAOK> addition, the data frequently contains errors. The appropriate RAOK> response to such errors is application-specific. Some RAOK> applications can simply discard unexpected or erroneous values RAOK> and continue processing. For other applications, however, errors RAOK> in the data can be the most interesting part of the data. RAOK> Understanding a new data source and producing a suitable parser RAOK> are crucial first steps in any use of such data. Unfortunately, RAOK> writing parsers for this kind of data is a difficult task, both RAOK> tedious and error-prone. It is complicated by lack of RAOK> documentation, convoluted encodings designed to save space, the RAOK> need to handle errors robustly, and the need to produce RAOK> efficient code to cope with the scale of the stream. Often, the RAOK> hard-won understanding of the data ends up embedded in parsing RAOK> code, making long-term maintenance difficult for the original RAOK> writer and sharing the knowledge with others nearly impossible. RAOK> The goal of the PADS project is to provide languages and tools RAOK> for simplifying data processing. We have a preliminary design of RAOK> a declarative data-description language, PADSL, expressive RAOK> enough to describe the data feeds we see at AT&T in practice, RAOK> including ASCII, binary, EBCDIC, Cobol, and mixed data formats. RAOK> From PADSL we generate a tunable C library with functions for RAOK> parsing, manipulating, and summarizing the data. In joint work RAOK> with Mary Fernandez and Ricardo Medel, we are working to RAOK> integrate PADS and XQuery to support declarative querying of RAOK> data sources with PADS descriptions. RAOK> -------------------- RAOK> The PADS project moved from AT&T to RAOK> http://pads.cs.tufts.edu/doc.html RAOK> I say this in all seriousness: if I had a need to process RAOK> GB of HL7 data, I would start by seeing if PADS was adequate RAOK> to describe it, and if so I'd write an HL7->Erlang data RAOK> translator in C or ML (as PADS has C and ML versions). RAOK> If not, I'd see what ideas I could steal from PADS. RAOK> Using a declarative data language to describe the message types RAOK> I was interested in would be an up-front cost, but it would RAOK> hugely simplify later maintenance. -- Best regards, Andrew mailto:andrew@REDACTED sent from a real computer From ok@REDACTED Wed Aug 9 02:02:37 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 9 Aug 2017 12:02:37 +1200 Subject: [erlang-questions] List Question In-Reply-To: <1523137591.20170809094022@Medical-Objects.com.au> References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> <45821730.20170808195252@Medical-Objects.com.au> <1523137591.20170809094022@Medical-Objects.com.au> Message-ID: On 9/08/17 11:40 AM, Andrew McIntyre wrote: > I am involved in standards process and there is a blog post here for > anyone interested: > > https://kb.medical-objects.com.au/display/PUB/HL7v2+parsing Ah. It turns out that I had misunderstood the function of the escape character. I thought you would represent "~" as "\~" but in fact it's \F\ | \R\ ~ \S\ ^ \T\ & \E\ \ \.br\ CR -- isn't uniformity wonderful? plus a bunch of other "free text formatting commands" written as escape sequences (which may explain \.br\) but not described in that page. There is also encoding of non-ASCII characters using HTML escapes (exact set not specified in that page). So there are several rounds of processing: - build tree using delimiters and ignoring escapes - maybe process escapes - maybe process HTML &#...; escapes - maybe do something with formatting commands. and dealing with data types comes somewhere between the third and fourth rounds. How hard is it to get hold of the actual standard? (I can get Z39.50 for free...) From andrew@REDACTED Wed Aug 9 02:09:58 2017 From: andrew@REDACTED (Andrew McIntyre) Date: Wed, 9 Aug 2017 10:09:58 +1000 Subject: [erlang-questions] List Question In-Reply-To: References: <1288613114.20170807221323@Medical-Objects.com.au> <35768330.vdyX1H0hul@changa> <654058386.20170807222931@Medical-Objects.com.au> <1627741.xJ36y4QYPI@changa> <978091051.20170807234652@Medical-Objects.com.au> <45821730.20170808195252@Medical-Objects.com.au> <1523137591.20170809094022@Medical-Objects.com.au> Message-ID: <156654716.20170809100958@Medical-Objects.com.au> Hello Richard, I think if you register with hl7.org you can get some for free. There is an Australian localization here: http://confluence.hl7australia.com/display/OO/Australian+Pathology+Messaging+-+Localisation+of+HL7+Version+2.4 Andrew Wednesday, August 9, 2017, 10:02:37 AM, you wrote: RAOK> On 9/08/17 11:40 AM, Andrew McIntyre wrote: >> I am involved in standards process and there is a blog post here for >> anyone interested: >> >> https://kb.medical-objects.com.au/display/PUB/HL7v2+parsing RAOK> Ah. It turns out that I had misunderstood the function of the RAOK> escape character. I thought you would represent "~" as "\~" RAOK> but in fact it's RAOK> \F\ | RAOK> \R\ ~ RAOK> \S\ ^ RAOK> \T\ & RAOK> \E\ \ RAOK> \.br\ CR -- isn't uniformity wonderful? RAOK> plus a bunch of other "free text formatting commands" RAOK> written as escape sequences (which may explain \.br\) RAOK> but not described in that page. RAOK> There is also encoding of non-ASCII characters using RAOK> HTML escapes (exact set not specified in that page). RAOK> So there are several rounds of processing: RAOK> - build tree using delimiters and ignoring escapes RAOK> - maybe process escapes RAOK> - maybe process HTML &#...; escapes RAOK> - maybe do something with formatting commands. RAOK> and dealing with data types comes somewhere between RAOK> the third and fourth rounds. RAOK> How hard is it to get hold of the actual standard? RAOK> (I can get Z39.50 for free...) -- Best regards, Andrew mailto:andrew@REDACTED sent from a real computer From aschultz@REDACTED Wed Aug 9 10:45:36 2017 From: aschultz@REDACTED (Andreas Schultz) Date: Wed, 9 Aug 2017 10:45:36 +0200 (CEST) Subject: [erlang-questions] Retrieving TLS Information After/On Connect In-Reply-To: <1D29362A-490C-4A6F-ADD2-06C9FF6B4E6E@gmail.com> References: <1D29362A-490C-4A6F-ADD2-06C9FF6B4E6E@gmail.com> Message-ID: <1588899556.873967.1502268336986.JavaMail.zimbra@tpip.net> Hi, ----- On Aug 8, 2017, at 9:18 PM, Ryan Auger wrote: > Hello, > I am using PSKs to connect to my VMQTT/EMQTT instance, and need to modify the > source code or write a plugin to set my clientid or username = psk-identity, > thus creating a ?TLS-PSK based authentication?. To be clear, my connection > using PSKs works, and I can connect and send encrypted messages using an > incredibly basic user_lookup_fun like so: > user_lookup(psk, _Username, UserState) -> > {ok, UserState}. I think you are misunderstanding the user_lookup/3 function. The UserName argument will contain the PSK client identity from the TLS handshake, the returned value should be `{ok, ShareSecret}`. The UserState argument can be used to point the user_lookup function to the place where the share secret is stored. It could be a map, a ets TID or the pid of a process knowing about the secret. Your sample is working, because the simplest implementation just stores the shared secret in UserState and simply returns that. For a real world use case that is obviously not going to be very useful. The real problem is after your TLS connection has been established, how to extract the PSK identity (or SRP username for that matter) from the TLS connection. A interface similar to ssl:peercert/1 to the SSL connection is currently missing. Adding such a API to ssl.erl is not very difficult, the real question is should it be peer_identity(#sslsocket{}) ->{ok, binary()} | {error, reason()}. or a more generic version like peer_identity(#sslsocket{}, Type :: 'certificate' | 'psk_identity' | 'srp_username' | 'whatever') ->{ok, Identity} | {error, reason()}. Regards Andreas > The problem here is that I have no reference to the calling process, and can?t > modify the ClientID or username, so I cannot authenticate users without sending > an additional username and password. If someone gained access to my private > key, they would be able to publish and subscribe to any topic because the > psk_identity is not currently used in authentication. > The way I see it, there are two options: > 1) Set the ClientID in user lookup -> this is impossible because I have no > reference to the calling process > 2) Find where the ClientID is set, use a function to get the psk-identity from > the listening process, and then set the ClientID = psk-identity > For option 2, my first goal is to just get the psk_identity from somewhere in > the code. I tried to use the erlang library function > ssl_connection:connection_information(Client#mqtt_client.client_pid)]) on the > client.connected hook, > But this process hangs and does not return any value. > Can anyone point me in the right direction for getting the TLS information from > within the code? > Otherwise, is there any way that I could go about option 1) without knowing > anything about the calling process? > Thank you in advance! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From gootik_2003@REDACTED Fri Aug 11 17:40:36 2017 From: gootik_2003@REDACTED (Sasan Hezarkhani) Date: Fri, 11 Aug 2017 15:40:36 +0000 Subject: [erlang-questions] This week in Erlang Aug 11 Message-ID: Hi there, It?s been a while since I last posted about this newsletter, I thought it would be a good reminder. Sorry if it is annoying and you already know about it. Medium post: https://medium.com/@gootik/this-week-in-erlang-aug-11-5300a0bc7e27 You can help/fix/suggest/feedback in Github now too! https://github.com/gootik/this-week-in-erlang Thank you, Sasan -------------- next part -------------- An HTML attachment was scrubbed... URL: From gordeev.vladimir.v@REDACTED Fri Aug 11 20:28:38 2017 From: gordeev.vladimir.v@REDACTED (Vladimir Gordeev) Date: Fri, 11 Aug 2017 21:28:38 +0300 Subject: [erlang-questions] This week in Erlang Aug 11 In-Reply-To: References: Message-ID: Great idea! Hope you won't run out of patience doing it. On Fri, Aug 11, 2017 at 6:40 PM, Sasan Hezarkhani wrote: > Hi there, > It?s been a while since I last posted about this newsletter, I thought it > would be a good reminder. Sorry if it is annoying and you already know > about it. > > Medium post: https://medium.com/@gootik/this-week-in-erlang- > aug-11-5300a0bc7e27 > You can help/fix/suggest/feedback in Github now too! https://github.com/ > gootik/this-week-in-erlang > > Thank you, > Sasan > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From khitai.pang@REDACTED Mon Aug 14 05:17:38 2017 From: khitai.pang@REDACTED (Khitai Pang) Date: Mon, 14 Aug 2017 03:17:38 +0000 Subject: [erlang-questions] gen_server: {bad_return_value, {error, {91, invalid_string}}} Message-ID: Hi list, On my CentOS7 server with esl-erlang-19.3.6-1.x86_64, a gen_server process mysteriously crashes. See the error info below. I can't figure out the reason of the error. Any hint would be appreciated. {error_report,<0.25443.97>, {<0.28918.97>,crash_report, [[{initial_call,{xeef_analyzer,init,[Argument__1]}}, {pid,<0.28918.97>}, {registered_name,xeef_analyzer_009}, {error_info, {exit, {bad_return_value,{error,{91,invalid_string}}}, [{gen_server,terminate,7,[{file,"gen_server.erl"},{line,812}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,247}]}]}}, {ancestors,[xeef_analyzer_sup,xeef_sup,<0.25310.97>]}, {messages,[{basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>}]}, {links, [#Port<0.2317500>,#Port<0.2317503>,<0.29009.97>,#Port<0.2317499>]}, {dictionary, [{rand_seed, {#{max => 18446744073709551615,next => #Fun, type => exs1024,uniform => #Fun, uniform_n => #Fun}, {[5529181756040335035,6774320120208346895, 4985297473998268992,12813505499546735856, 3059599059795068553,3134963973831305956, 5508046504858988103,3821329447243874404], [6547380368325771448,16036002885223559991, 13982350927889545678,14401258729428095242, 6347438184782916914,10561027033563628251, 10847483533189471003,3486622809589930158]}}}]}, {trap_exit,true}, {status,running}, {heap_size,987}, {stack_size,27}, {reductions,139726}], []]}} {error_report,<0.25443.97>, {<0.29009.97>,supervisor_report, [{supervisor,{local,xeef_analyzer_sup}}, {errorContext,child_terminated}, {reason,{bad_return_value,{error,{91,invalid_string}}}}, {offender, [{pid,<0.28918.97>}, {id,xeef_analyzer}, {mfargs, {xeef_analyzer,start_link, [{sandbox,{<0.28962.97>,<<"sanbox_queue">>},none,true}]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,worker}]}]}} Best wishes, Khitai From hans.r.nilsson@REDACTED Mon Aug 14 12:29:33 2017 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Mon, 14 Aug 2017 12:29:33 +0200 Subject: [erlang-questions] SSH 4.5 bug ? In-Reply-To: <1500979313.13455.1.camel@free.fr> References: <1500979313.13455.1.camel@free.fr> Message-ID: <550d7971-68b2-29ee-ff5c-ecc8b01516d7@ericsson.com> Can you try $ ssh -p 10022 -vvv localhost to get more details? -Hans On 07/25/2017 12:41 PM, Jean Parpaillon wrote: > $ ssh -p 10022 -v localhost From raimo+erlang-questions@REDACTED Mon Aug 14 13:34:05 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 14 Aug 2017 13:34:05 +0200 Subject: [erlang-questions] gen_server: {bad_return_value, {error, {91, invalid_string}}} In-Reply-To: References: Message-ID: <20170814113405.GA3042@erix.ericsson.se> On Mon, Aug 14, 2017 at 03:17:38AM +0000, Khitai Pang wrote: > Hi list, > > On my CentOS7 server with esl-erlang-19.3.6-1.x86_64, a gen_server > process mysteriously crashes. See the error info below. I can't figure > out the reason of the error. Any hint would be appreciated. > > {error_report,<0.25443.97>, > {<0.28918.97>,crash_report, > [[{initial_call,{xeef_analyzer,init,[Argument__1]}}, > {pid,<0.28918.97>}, > {registered_name,xeef_analyzer_009}, > {error_info, > {exit, > {bad_return_value,{error,{91,invalid_string}}}, It seems your gen_server with registered name xeef_analyzer_009 started from xeef_analyzer:init/1 has returned {error,{91,invalid_string}} from one of its callbacks. > [{gen_server,terminate,7,[{file,"gen_server.erl"},{line,812}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,247}]}]}}, > {ancestors,[xeef_analyzer_sup,xeef_sup,<0.25310.97>]}, > {messages,[{basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>}]}, > {links, > [#Port<0.2317500>,#Port<0.2317503>,<0.29009.97>,#Port<0.2317499>]}, > {dictionary, > [{rand_seed, > {#{max => 18446744073709551615,next => > #Fun, > type => exs1024,uniform => #Fun, > uniform_n => #Fun}, > {[5529181756040335035,6774320120208346895, > 4985297473998268992,12813505499546735856, > 3059599059795068553,3134963973831305956, > 5508046504858988103,3821329447243874404], > [6547380368325771448,16036002885223559991, > 13982350927889545678,14401258729428095242, > 6347438184782916914,10561027033563628251, > 10847483533189471003,3486622809589930158]}}}]}, > {trap_exit,true}, > {status,running}, > {heap_size,987}, > {stack_size,27}, > {reductions,139726}], > []]}} > > > {error_report,<0.25443.97>, > {<0.29009.97>,supervisor_report, > [{supervisor,{local,xeef_analyzer_sup}}, > {errorContext,child_terminated}, > {reason,{bad_return_value,{error,{91,invalid_string}}}}, > {offender, > [{pid,<0.28918.97>}, > {id,xeef_analyzer}, > {mfargs, > {xeef_analyzer,start_link, > [{sandbox,{<0.28962.97>,<<"sanbox_queue">>},none,true}]}}, > {restart_type,permanent}, > {shutdown,5000}, > {child_type,worker}]}]}} > > > > Best wishes, > Khitai > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From attila.r.nohl@REDACTED Mon Aug 14 13:54:28 2017 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Mon, 14 Aug 2017 13:54:28 +0200 Subject: [erlang-questions] gen_server: {bad_return_value, {error, {91, invalid_string}}} In-Reply-To: References: Message-ID: Hello! I'd check the code that handles the {basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>} message, because this was the last received message. Also my intuition is that somewhere your code expects a list of strings, but gets a simple string, and when it checks the first element, it finds 91 (probably the '[' character) instead of a string. 2017-08-14 5:17 GMT+02:00 Khitai Pang : > Hi list, > > On my CentOS7 server with esl-erlang-19.3.6-1.x86_64, a gen_server > process mysteriously crashes. See the error info below. I can't figure > out the reason of the error. Any hint would be appreciated. > > {error_report,<0.25443.97>, > {<0.28918.97>,crash_report, > [[{initial_call,{xeef_analyzer,init,[Argument__1]}}, > {pid,<0.28918.97>}, > {registered_name,xeef_analyzer_009}, > {error_info, > {exit, > {bad_return_value,{error,{91,invalid_string}}}, > [{gen_server,terminate,7,[{file,"gen_server.erl"},{line,812}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,247}]}]}}, > {ancestors,[xeef_analyzer_sup,xeef_sup,<0.25310.97>]}, > {messages,[{basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>}]}, > {links, > [#Port<0.2317500>,#Port<0.2317503>,<0.29009.97>,#Port<0.2317499>]}, > {dictionary, > [{rand_seed, > {#{max => 18446744073709551615,next => > #Fun, > type => exs1024,uniform => #Fun, > uniform_n => #Fun}, > {[5529181756040335035,6774320120208346895, > 4985297473998268992,12813505499546735856, > 3059599059795068553,3134963973831305956, > 5508046504858988103,3821329447243874404], > [6547380368325771448,16036002885223559991, > 13982350927889545678,14401258729428095242, > 6347438184782916914,10561027033563628251, > 10847483533189471003,3486622809589930158]}}}]}, > {trap_exit,true}, > {status,running}, > {heap_size,987}, > {stack_size,27}, > {reductions,139726}], > []]}} > > > {error_report,<0.25443.97>, > {<0.29009.97>,supervisor_report, > [{supervisor,{local,xeef_analyzer_sup}}, > {errorContext,child_terminated}, > {reason,{bad_return_value,{error,{91,invalid_string}}}}, > {offender, > [{pid,<0.28918.97>}, > {id,xeef_analyzer}, > {mfargs, > {xeef_analyzer,start_link, > [{sandbox,{<0.28962.97>,<<"sanbox_queue">>},none,true}]}}, > {restart_type,permanent}, > {shutdown,5000}, > {child_type,worker}]}]}} > > > > Best wishes, > Khitai > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From raimo+erlang-questions@REDACTED Mon Aug 14 15:36:47 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 14 Aug 2017 15:36:47 +0200 Subject: [erlang-questions] Erlang 20 and crypto:rand_uniform/2 In-Reply-To: References: <20170517001632.841B3994@m0087794.ppops.net> <20170517075122.GA68388@erix.ericsson.se> Message-ID: <20170814133647.GA21830@erix.ericsson.se> On Tue, Aug 08, 2017 at 11:10:33AM -0700, Pierre Fenoll wrote: > I am sorry but nothing of what you said is even close to a good reason to > not provide a /2 providing the same API as crypto:rand_uniform/2. As I see it there is so far no good reason _for_ implementing rand:uniform/2, and there are at least 2 possible variants: rand:uniform(Min, Max) -> Min + rand:uniform(Max - Min). rand:uniform(Min, Range) -> Min + rand:uniform(Range). The current API is minimal, makes both easy and does not choose between them. Small APIs are in general more beautiful and easier to maintain. Best regards -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From khitai.pang@REDACTED Tue Aug 15 05:13:19 2017 From: khitai.pang@REDACTED (Khitai Pang) Date: Tue, 15 Aug 2017 03:13:19 +0000 Subject: [erlang-questions] gen_server: {bad_return_value, {error, {91, invalid_string}}} In-Reply-To: References: Message-ID: Hi Raimo, Attila, I have found out the cause: the handle_info callback of my gen_server failed to decode a json string: {Json} = jiffy:decode(Msg), the above code generates {error, {91, invalid_string}}, and I believe the error is caught by OTP gen_server code, and somehow the terminate() function in OTP gen_server code complains about {bad_return_value, {error, {91, invalid_string}}}. Now I know the root cause of the problem but I don't understand why it ends up in an error of gen_server:terminate(). Best wishes, Khitai On 2017/8/14 19:54, Attila Rajmund Nohl wrote: > Hello! > > I'd check the code that handles the > {basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>} message, > because this was the last received message. Also my intuition is that > somewhere your code expects a list of strings, but gets a simple > string, and when it checks the first element, it finds 91 (probably > the '[' character) instead of a string. > > 2017-08-14 5:17 GMT+02:00 Khitai Pang : >> Hi list, >> >> On my CentOS7 server with esl-erlang-19.3.6-1.x86_64, a gen_server >> process mysteriously crashes. See the error info below. I can't figure >> out the reason of the error. Any hint would be appreciated. >> >> {error_report,<0.25443.97>, >> {<0.28918.97>,crash_report, >> [[{initial_call,{xeef_analyzer,init,[Argument__1]}}, >> {pid,<0.28918.97>}, >> {registered_name,xeef_analyzer_009}, >> {error_info, >> {exit, >> {bad_return_value,{error,{91,invalid_string}}}, >> [{gen_server,terminate,7,[{file,"gen_server.erl"},{line,812}]}, >> {proc_lib,init_p_do_apply,3, >> [{file,"proc_lib.erl"},{line,247}]}]}}, >> {ancestors,[xeef_analyzer_sup,xeef_sup,<0.25310.97>]}, >> {messages,[{basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>}]}, >> {links, >> [#Port<0.2317500>,#Port<0.2317503>,<0.29009.97>,#Port<0.2317499>]}, >> {dictionary, >> [{rand_seed, >> {#{max => 18446744073709551615,next => >> #Fun, >> type => exs1024,uniform => #Fun, >> uniform_n => #Fun}, >> {[5529181756040335035,6774320120208346895, >> 4985297473998268992,12813505499546735856, >> 3059599059795068553,3134963973831305956, >> 5508046504858988103,3821329447243874404], >> [6547380368325771448,16036002885223559991, >> 13982350927889545678,14401258729428095242, >> 6347438184782916914,10561027033563628251, >> 10847483533189471003,3486622809589930158]}}}]}, >> {trap_exit,true}, >> {status,running}, >> {heap_size,987}, >> {stack_size,27}, >> {reductions,139726}], >> []]}} >> >> >> {error_report,<0.25443.97>, >> {<0.29009.97>,supervisor_report, >> [{supervisor,{local,xeef_analyzer_sup}}, >> {errorContext,child_terminated}, >> {reason,{bad_return_value,{error,{91,invalid_string}}}}, >> {offender, >> [{pid,<0.28918.97>}, >> {id,xeef_analyzer}, >> {mfargs, >> {xeef_analyzer,start_link, >> [{sandbox,{<0.28962.97>,<<"sanbox_queue">>},none,true}]}}, >> {restart_type,permanent}, >> {shutdown,5000}, >> {child_type,worker}]}]}} >> >> >> >> Best wishes, >> Khitai >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From ali.sabil@REDACTED Tue Aug 15 11:09:05 2017 From: ali.sabil@REDACTED (Ali Sabil) Date: Tue, 15 Aug 2017 11:09:05 +0200 Subject: [erlang-questions] gen_server: {bad_return_value, {error, {91, invalid_string}}} In-Reply-To: References: Message-ID: Hi Khitai, That's because jiffy (mis)uses erlang:throw. Best, Ali On Tue, Aug 15, 2017 at 5:13 AM, Khitai Pang wrote: > Hi Raimo, Attila, > > I have found out the cause: the handle_info callback of my gen_server > failed to decode a json string: > > {Json} = jiffy:decode(Msg), > > the above code generates {error, {91, invalid_string}}, and I believe > the error is caught by OTP gen_server code, and somehow the terminate() > function in OTP gen_server code complains about {bad_return_value, > {error, {91, invalid_string}}}. Now I know the root cause of the > problem but I don't understand why it ends up in an error of > gen_server:terminate(). > > > Best wishes, > Khitai > > On 2017/8/14 19:54, Attila Rajmund Nohl wrote: > > Hello! > > > > I'd check the code that handles the > > {basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>} message, > > because this was the last received message. Also my intuition is that > > somewhere your code expects a list of strings, but gets a simple > > string, and when it checks the first element, it finds 91 (probably > > the '[' character) instead of a string. > > > > 2017-08-14 5:17 GMT+02:00 Khitai Pang : > >> Hi list, > >> > >> On my CentOS7 server with esl-erlang-19.3.6-1.x86_64, a gen_server > >> process mysteriously crashes. See the error info below. I can't figure > >> out the reason of the error. Any hint would be appreciated. > >> > >> {error_report,<0.25443.97>, > >> {<0.28918.97>,crash_report, > >> [[{initial_call,{xeef_analyzer,init,[Argument__1]}}, > >> {pid,<0.28918.97>}, > >> {registered_name,xeef_analyzer_009}, > >> {error_info, > >> {exit, > >> {bad_return_value,{error,{91,invalid_string}}}, > >> [{gen_server,terminate,7,[{file,"gen_server.erl"},{line,812}]}, > >> {proc_lib,init_p_do_apply,3, > >> [{file,"proc_lib.erl"},{line,247}]}]}}, > >> {ancestors,[xeef_analyzer_sup,xeef_sup,<0.25310.97>]}, > >> {messages,[{basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>}]}, > >> {links, > >> [#Port<0.2317500>,#Port<0.2317503>,<0.29009.97>,#Port<0.2317499>]}, > >> {dictionary, > >> [{rand_seed, > >> {#{max => 18446744073709551615,next => > >> #Fun, > >> type => exs1024,uniform => #Fun, > >> uniform_n => #Fun}, > >> {[5529181756040335035,6774320120208346895, > >> 4985297473998268992,12813505499546735856, > >> 3059599059795068553,3134963973831305956, > >> 5508046504858988103,3821329447243874404], > >> [6547380368325771448,16036002885223559991, > >> 13982350927889545678,14401258729428095242, > >> 6347438184782916914,10561027033563628251, > >> 10847483533189471003,3486622809589930158]}}}]}, > >> {trap_exit,true}, > >> {status,running}, > >> {heap_size,987}, > >> {stack_size,27}, > >> {reductions,139726}], > >> []]}} > >> > >> > >> {error_report,<0.25443.97>, > >> {<0.29009.97>,supervisor_report, > >> [{supervisor,{local,xeef_analyzer_sup}}, > >> {errorContext,child_terminated}, > >> {reason,{bad_return_value,{error,{91,invalid_string}}}}, > >> {offender, > >> [{pid,<0.28918.97>}, > >> {id,xeef_analyzer}, > >> {mfargs, > >> {xeef_analyzer,start_link, > >> [{sandbox,{<0.28962.97>,<<"sanbox_queue">>},none,true}]}}, > >> {restart_type,permanent}, > >> {shutdown,5000}, > >> {child_type,worker}]}]}} > >> > >> > >> > >> Best wishes, > >> Khitai > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Tue Aug 15 11:17:08 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 15 Aug 2017 11:17:08 +0200 Subject: [erlang-questions] gen_server: {bad_return_value, {error, {91, invalid_string}}} In-Reply-To: References: Message-ID: <20170815091708.GA19231@erix.ericsson.se> On Tue, Aug 15, 2017 at 11:09:05AM +0200, Ali Sabil wrote: > Hi Khitai, > > That's because jiffy (mis)uses erlang:throw. If it is so that jiffy:decode(Msg) throws {error,{91,invalid_string}}, then it happens to be one way to return from a callback i.e to throw the value. So, yes - if so jiffy misuses erlang:throw. Jiffy should use erlang:error if it is an error. / Raimo > > Best, > Ali > > On Tue, Aug 15, 2017 at 5:13 AM, Khitai Pang > wrote: > > > Hi Raimo, Attila, > > > > I have found out the cause: the handle_info callback of my gen_server > > failed to decode a json string: > > > > {Json} = jiffy:decode(Msg), > > > > the above code generates {error, {91, invalid_string}}, and I believe > > the error is caught by OTP gen_server code, and somehow the terminate() > > function in OTP gen_server code complains about {bad_return_value, > > {error, {91, invalid_string}}}. Now I know the root cause of the > > problem but I don't understand why it ends up in an error of > > gen_server:terminate(). > > > > > > Best wishes, > > Khitai > > > > On 2017/8/14 19:54, Attila Rajmund Nohl wrote: > > > Hello! > > > > > > I'd check the code that handles the > > > {basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>} message, > > > because this was the last received message. Also my intuition is that > > > somewhere your code expects a list of strings, but gets a simple > > > string, and when it checks the first element, it finds 91 (probably > > > the '[' character) instead of a string. > > > > > > 2017-08-14 5:17 GMT+02:00 Khitai Pang : > > >> Hi list, > > >> > > >> On my CentOS7 server with esl-erlang-19.3.6-1.x86_64, a gen_server > > >> process mysteriously crashes. See the error info below. I can't figure > > >> out the reason of the error. Any hint would be appreciated. > > >> > > >> {error_report,<0.25443.97>, > > >> {<0.28918.97>,crash_report, > > >> [[{initial_call,{xeef_analyzer,init,[Argument__1]}}, > > >> {pid,<0.28918.97>}, > > >> {registered_name,xeef_analyzer_009}, > > >> {error_info, > > >> {exit, > > >> {bad_return_value,{error,{91,invalid_string}}}, > > >> [{gen_server,terminate,7,[{file,"gen_server.erl"},{line,812}]}, > > >> {proc_lib,init_p_do_apply,3, > > >> [{file,"proc_lib.erl"},{line,247}]}]}}, > > >> {ancestors,[xeef_analyzer_sup,xeef_sup,<0.25310.97>]}, > > >> {messages,[{basic.cancel_ok,<<"amq.ctag-YRIyK1Kv0i93rkLbD9Zo8w">>}]}, > > >> {links, > > >> [#Port<0.2317500>,#Port<0.2317503>,<0.29009.97>,#Port<0.2317499>]}, > > >> {dictionary, > > >> [{rand_seed, > > >> {#{max => 18446744073709551615,next => > > >> #Fun, > > >> type => exs1024,uniform => #Fun, > > >> uniform_n => #Fun}, > > >> {[5529181756040335035,6774320120208346895, > > >> 4985297473998268992,12813505499546735856, > > >> 3059599059795068553,3134963973831305956, > > >> 5508046504858988103,3821329447243874404], > > >> [6547380368325771448,16036002885223559991, > > >> 13982350927889545678,14401258729428095242, > > >> 6347438184782916914,10561027033563628251, > > >> 10847483533189471003,3486622809589930158]}}}]}, > > >> {trap_exit,true}, > > >> {status,running}, > > >> {heap_size,987}, > > >> {stack_size,27}, > > >> {reductions,139726}], > > >> []]}} > > >> > > >> > > >> {error_report,<0.25443.97>, > > >> {<0.29009.97>,supervisor_report, > > >> [{supervisor,{local,xeef_analyzer_sup}}, > > >> {errorContext,child_terminated}, > > >> {reason,{bad_return_value,{error,{91,invalid_string}}}}, > > >> {offender, > > >> [{pid,<0.28918.97>}, > > >> {id,xeef_analyzer}, > > >> {mfargs, > > >> {xeef_analyzer,start_link, > > >> [{sandbox,{<0.28962.97>,<<"sanbox_queue">>},none,true}]}}, > > >> {restart_type,permanent}, > > >> {shutdown,5000}, > > >> {child_type,worker}]}]}} > > >> > > >> > > >> > > >> Best wishes, > > >> Khitai > > >> _______________________________________________ > > >> erlang-questions mailing list > > >> erlang-questions@REDACTED > > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From Catenacci@REDACTED Tue Aug 15 12:14:25 2017 From: Catenacci@REDACTED (Onorio Catenacci) Date: Tue, 15 Aug 2017 06:14:25 -0400 Subject: [erlang-questions] [ANN] Erlang 20 is on Chocolatey NuGet Message-ID: For those who care, I've updated the Erlang package on Chocolatey NuGet to OTP 20.0 https://chocolatey.org/packages/erlang -- Onorio Catenacci http://onor.io http://www.google.com/+OnorioCatenacci -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Wed Aug 16 13:20:56 2017 From: gerhard@REDACTED (Gerhard Lazu) Date: Wed, 16 Aug 2017 12:20:56 +0100 Subject: [erlang-questions] OTP 20.0.2 segfault at 5c ip in beam.smp Message-ID: Hi, A RabbitMQ 3.6.11-rc.3 node crashed when OTP 20.0.2 segfaulted. This is the only information that we have about the Erlang VM itself: Aug 16 04:08:59 localhost kernel: [61139.180699] 2_scheduler[15162]: segfault at 5c ip 00000000005ba760 sp 00007f8fea1ffc40 err or 4 in beam.smp[400000+333000] This is the last line logged by RMQ, 2 seconds before the Erlang VM segfault: =INFO REPORT==== 16-Aug-2017::04:08:57 === closing AMQP connection <0.24230.9> (10.0.16.35:42562 -> 10.0.16.23:5672, vhost: '/', user: 'admin') beam.smp runs with the following flags: /var/vcap/packages/erlang-20.0.2/lib/erlang/erts-9.0.2/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /var/vcap/packages/erlang-20.0.2/lib/erlang -progname erl -- -home /home/vcap -- -pa /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/ebin -noshell -noinput -s rabbit boot -sname rabbit@REDACTED -boot start_sasl -config /var/vcap/jobs/rabbitmq-server/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/vcap/sys/log/ rabbitmq-server/rabbit@REDACTED"} -rabbit sasl_error_logger {file,"/var/vcap/sys/log/rabbitmq-server/rabbit@REDACTED"} -rabbit enabled_plugins_file "/var/vcap/jobs/rabbitmq- server/packages/rabbitmq-server/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/plugins" -rabbit plugins_expand_dir "/var/vcap/store/rabbitmq- server/mnesia/rabbit@REDACTED" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/vcap/store/rabbitmq-server/mnesia/rabbit@REDACTED" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 We are running on: 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 x86_64 GNU/Linux When this happens again, how do I capture more debugging information? I would like to file an OTP bug report, but I'm not sure that we have enough information. Apparently, this was reported at least once before, against OTP 18.1: https://github.com/rabbitmq/rabbitmq-server/issues/459 Thank you, Gerhard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Aug 16 14:42:39 2017 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 16 Aug 2017 14:42:39 +0200 Subject: [erlang-questions] OTP 20.0.2 segfault at 5c ip in beam.smp In-Reply-To: References: Message-ID: Hello, In order to debug this we need access to a core file generated from the crash and also the beam.smp binary that was used to run it. Lukas On Wed, Aug 16, 2017 at 1:20 PM, Gerhard Lazu wrote: > Hi, > > A RabbitMQ 3.6.11-rc.3 node crashed when OTP 20.0.2 segfaulted. This is > the only information that we have about the Erlang VM itself: > > Aug 16 04:08:59 localhost kernel: [61139.180699] 2_scheduler[15162]: > segfault at 5c ip 00000000005ba760 sp 00007f8fea1ffc40 err > or 4 in beam.smp[400000+333000] > > This is the last line logged by RMQ, 2 seconds before the Erlang VM > segfault: > > =INFO REPORT==== 16-Aug-2017::04:08:57 === closing AMQP connection > <0.24230.9> (10.0.16.35:42562 -> 10.0.16.23:5672, vhost: '/', user: > 'admin') > > beam.smp runs with the following flags: > > /var/vcap/packages/erlang-20.0.2/lib/erlang/erts-9.0.2/bin/beam.smp > -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root > /var/vcap/packages/erlang-20.0.2/lib/erlang -progname erl -- -home > /home/vcap -- -pa /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/ebin > -noshell -noinput -s rabbit boot -sname rabbit@REDACTED -boot > start_sasl -config /var/vcap/jobs/rabbitmq-server/rabbitmq -kernel > inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl > sasl_error_logger false -rabbit error_logger {file,"/var/vcap/sys/log/rabbi > tmq-server/rabbit@REDACTED"} -rabbit sasl_error_logger > {file,"/var/vcap/sys/log/rabbitmq-server/rabbit@REDACTED"} > -rabbit enabled_plugins_file "/var/vcap/jobs/rabbitmq-serve > r/packages/rabbitmq-server/etc/rabbitmq/enabled_plugins" -rabbit > plugins_dir "/var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/plugins" > -rabbit plugins_expand_dir "/var/vcap/store/rabbitmq-serv > er/mnesia/rabbit@REDACTED" -os_mon start_cpu_sup > false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir > "/var/vcap/store/rabbitmq-server/mnesia/rabbit@REDACTED" -kernel > inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 > > We are running on: > > 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 > x86_64 GNU/Linux > > When this happens again, how do I capture more debugging information? I > would like to file an OTP bug report, but I'm not sure that we have enough > information. > > Apparently, this was reported at least once before, against OTP 18.1: > https://github.com/rabbitmq/rabbitmq-server/issues/459 > > Thank you, Gerhard. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Wed Aug 16 15:53:04 2017 From: codewiget95@REDACTED (code wiget) Date: Wed, 16 Aug 2017 09:53:04 -0400 Subject: [erlang-questions] Proper way to run C code without crashing the VM Message-ID: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Hello, I have been reading about NIF?s here: http://erlang.org/doc/tutorial/nif.html but it seems like they are incredibly unsafe: ?it is also the least safe, because a crash in a NIF brings the emulator down too?. The problem here is huge, ideally I can?t ever have my Erlang VM close. What solution do you Erlangers use when you have to run low level code? My goal is to send a sort of ?GET? request with a UUID and I will receive back that UUID along with some binary values. My C code works with this, but I can?t jeopardize the entire server crashing. Thank you for your help! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Wed Aug 16 15:58:46 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Wed, 16 Aug 2017 15:58:46 +0200 Subject: [erlang-questions] Proper way to run C code without crashing the VM In-Reply-To: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> References: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Message-ID: When the stability is important you have several choices: 1. Not use a NIF, write your logic in Erlang. Big industrial Erlang users may prefer this, for better debuggability and predictability at the cost of performance. 2. Write a really safe NIF which doesn't crash. Use defensive programming approach. Use safe language like Rust. 3. Separate your C code from Erlang node, write a port ? a console application which accepts input in stdin and prints results to stdout, then use Erlang open port function to start it and port commands to communicate with it. 2017-08-16 15:53 GMT+02:00 code wiget : > Hello, > > I have been reading about NIF?s here: http://erlang.org/doc/ > tutorial/nif.html but it seems like they are incredibly unsafe: ?it is > also the least safe, because a crash in a NIF brings the emulator down > too?. The problem here is huge, ideally I can?t ever have my Erlang VM > close. > > What solution do you Erlangers use when you have to run low level code? My > goal is to send a sort of ?GET? request with a UUID and I will receive back > that UUID along with some binary values. My C code works with this, but I > can?t jeopardize the entire server crashing. > > Thank you for your help! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Wed Aug 16 17:00:21 2017 From: gerhard@REDACTED (Gerhard Lazu) Date: Wed, 16 Aug 2017 16:00:21 +0100 Subject: [erlang-questions] OTP 20.0.2 segfault at 5c ip in beam.smp In-Reply-To: References: Message-ID: The crash dump was never generated because the Erlang VM segfaulted. Is there a way to force extra debugging output in segfault scenarios? On Wed, Aug 16, 2017 at 1:42 PM, Lukas Larsson wrote: > Hello, > > In order to debug this we need access to a core file generated from the > crash and also the beam.smp binary that was used to run it. > > Lukas > > On Wed, Aug 16, 2017 at 1:20 PM, Gerhard Lazu wrote: > >> Hi, >> >> A RabbitMQ 3.6.11-rc.3 node crashed when OTP 20.0.2 segfaulted. This is >> the only information that we have about the Erlang VM itself: >> >> Aug 16 04:08:59 localhost kernel: [61139.180699] 2_scheduler[15162]: >> segfault at 5c ip 00000000005ba760 sp 00007f8fea1ffc40 err >> or 4 in beam.smp[400000+333000] >> >> This is the last line logged by RMQ, 2 seconds before the Erlang VM >> segfault: >> >> =INFO REPORT==== 16-Aug-2017::04:08:57 === closing AMQP connection >> <0.24230.9> (10.0.16.35:42562 -> 10.0.16.23:5672, vhost: '/', user: >> 'admin') >> >> beam.smp runs with the following flags: >> >> /var/vcap/packages/erlang-20.0.2/lib/erlang/erts-9.0.2/bin/beam.smp >> -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root >> /var/vcap/packages/erlang-20.0.2/lib/erlang -progname erl -- -home >> /home/vcap -- -pa /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/ebin >> -noshell -noinput -s rabbit boot -sname rabbit@REDACTED -boot >> start_sasl -config /var/vcap/jobs/rabbitmq-server/rabbitmq -kernel >> inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl >> sasl_error_logger false -rabbit error_logger {file,"/var/vcap/sys/log/rabbi >> tmq-server/rabbit@REDACTED"} -rabbit sasl_error_logger >> {file,"/var/vcap/sys/log/rabbitmq-server/rabbit@REDACTED"} >> -rabbit enabled_plugins_file "/var/vcap/jobs/rabbitmq-serve >> r/packages/rabbitmq-server/etc/rabbitmq/enabled_plugins" -rabbit >> plugins_dir "/var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/plugins" >> -rabbit plugins_expand_dir "/var/vcap/store/rabbitmq-serv >> er/mnesia/rabbit@REDACTED" -os_mon start_cpu_sup >> false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir >> "/var/vcap/store/rabbitmq-server/mnesia/rabbit@REDACTED" -kernel >> inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 >> >> We are running on: >> >> 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 >> x86_64 GNU/Linux >> >> When this happens again, how do I capture more debugging information? I >> would like to file an OTP bug report, but I'm not sure that we have enough >> information. >> >> Apparently, this was reported at least once before, against OTP 18.1: >> https://github.com/rabbitmq/rabbitmq-server/issues/459 >> >> Thank you, Gerhard. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed Aug 16 17:05:41 2017 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 16 Aug 2017 16:05:41 +0100 Subject: [erlang-questions] Proper way to run C code without crashing the VM In-Reply-To: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> References: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Message-ID: On 16 August 2017 at 14:53, code wiget wrote: > I have been reading about NIF?s here: > http://erlang.org/doc/tutorial/nif.html but it seems like they are > incredibly unsafe: ?it is also the least safe, because a crash in a NIF > brings the emulator down too?. They're not "incredibly unsafe". There's nothing inherently unstable about a NIF. It's precisely as stable (or unstable) as the underlying implementation. It's just that it's running inside the BEAM, which means that *if* it crashes, it's going to crash the whole Erlang node. So: if you think you can write reliable C, go for it. If not, then you need to look at other options. See Dmytro's email. He skipped "C Node" as an option, incidentally. Another option is to host the NIF in another Erlang node, and use Erlang distribution to talk to it. From dmytro.lytovchenko@REDACTED Wed Aug 16 17:07:42 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Wed, 16 Aug 2017 17:07:42 +0200 Subject: [erlang-questions] Proper way to run C code without crashing the VM In-Reply-To: References: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Message-ID: Just an assumption, possibly a false assumption. If they already have a NIF, it means networked communication would possibly be slower than they expect. 2017-08-16 17:05 GMT+02:00 Roger Lipscombe : > On 16 August 2017 at 14:53, code wiget wrote: > > I have been reading about NIF?s here: > > http://erlang.org/doc/tutorial/nif.html but it seems like they are > > incredibly unsafe: ?it is also the least safe, because a crash in a NIF > > brings the emulator down too?. > > They're not "incredibly unsafe". There's nothing inherently unstable > about a NIF. It's precisely as stable (or unstable) as the underlying > implementation. It's just that it's running inside the BEAM, which > means that *if* it crashes, it's going to crash the whole Erlang node. > > So: if you think you can write reliable C, go for it. If not, then you > need to look at other options. See Dmytro's email. He skipped "C Node" > as an option, incidentally. Another option is to host the NIF in > another Erlang node, and use Erlang distribution to talk to it. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Aug 16 17:10:43 2017 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 16 Aug 2017 17:10:43 +0200 Subject: [erlang-questions] OTP 20.0.2 segfault at 5c ip in beam.smp In-Reply-To: References: Message-ID: You need to instruct the operating system to save a core dump, http://man7.org/linux/man-pages/man5/core.5.html. Some operating systems can be configured to dump parts of the core dump to the syslog when a segfault happens, which can be useful but a full core dump contains much more information so is a lot more useful. Lukas On Wed, Aug 16, 2017 at 5:00 PM, Gerhard Lazu wrote: > The crash dump was never generated because the Erlang VM segfaulted. > > Is there a way to force extra debugging output in segfault scenarios? > > On Wed, Aug 16, 2017 at 1:42 PM, Lukas Larsson wrote: > >> Hello, >> >> In order to debug this we need access to a core file generated from the >> crash and also the beam.smp binary that was used to run it. >> >> Lukas >> >> On Wed, Aug 16, 2017 at 1:20 PM, Gerhard Lazu wrote: >> >>> Hi, >>> >>> A RabbitMQ 3.6.11-rc.3 node crashed when OTP 20.0.2 segfaulted. This is >>> the only information that we have about the Erlang VM itself: >>> >>> Aug 16 04:08:59 localhost kernel: [61139.180699] 2_scheduler[15162]: >>> segfault at 5c ip 00000000005ba760 sp 00007f8fea1ffc40 err >>> or 4 in beam.smp[400000+333000] >>> >>> This is the last line logged by RMQ, 2 seconds before the Erlang VM >>> segfault: >>> >>> =INFO REPORT==== 16-Aug-2017::04:08:57 === closing AMQP connection >>> <0.24230.9> (10.0.16.35:42562 -> 10.0.16.23:5672, vhost: '/', user: >>> 'admin') >>> >>> beam.smp runs with the following flags: >>> >>> /var/vcap/packages/erlang-20.0.2/lib/erlang/erts-9.0.2/bin/beam.smp >>> -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root >>> /var/vcap/packages/erlang-20.0.2/lib/erlang -progname erl -- -home >>> /home/vcap -- -pa /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/ebin >>> -noshell -noinput -s rabbit boot -sname rabbit@REDACTED -boot >>> start_sasl -config /var/vcap/jobs/rabbitmq-server/rabbitmq -kernel >>> inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl >>> sasl_error_logger false -rabbit error_logger {file,"/var/vcap/sys/log/rabbi >>> tmq-server/rabbit@REDACTED"} -rabbit sasl_error_logger >>> {file,"/var/vcap/sys/log/rabbitmq-server/rabbit@REDACTED"} >>> -rabbit enabled_plugins_file "/var/vcap/jobs/rabbitmq-serve >>> r/packages/rabbitmq-server/etc/rabbitmq/enabled_plugins" -rabbit >>> plugins_dir "/var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/plugins" >>> -rabbit plugins_expand_dir "/var/vcap/store/rabbitmq-serv >>> er/mnesia/rabbit@REDACTED" -os_mon start_cpu_sup >>> false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir >>> "/var/vcap/store/rabbitmq-server/mnesia/rabbit@REDACTED" -kernel >>> inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 >>> >>> We are running on: >>> >>> 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC >>> 2017 x86_64 GNU/Linux >>> >>> When this happens again, how do I capture more debugging information? I >>> would like to file an OTP bug report, but I'm not sure that we have enough >>> information. >>> >>> Apparently, this was reported at least once before, against OTP 18.1: >>> https://github.com/rabbitmq/rabbitmq-server/issues/459 >>> >>> Thank you, Gerhard. >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Wed Aug 16 17:11:01 2017 From: codewiget95@REDACTED (code wiget) Date: Wed, 16 Aug 2017 11:11:01 -0400 Subject: [erlang-questions] Proper way to run C code without crashing the VM In-Reply-To: References: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Message-ID: <448778C5-81F2-45DD-ACA9-1E81F7FFBE2B@gmail.com> That is an interesting note about running it on another node and using it to communicate. For this though, I would then have to monitor when/if the node goes down as well . > On Aug 16, 2017, at 11:07 AM, Dmytro Lytovchenko wrote: > > Just an assumption, possibly a false assumption. > If they already have a NIF, it means networked communication would possibly be slower than they expect. > > 2017-08-16 17:05 GMT+02:00 Roger Lipscombe >: > On 16 August 2017 at 14:53, code wiget > wrote: > > I have been reading about NIF?s here: > > http://erlang.org/doc/tutorial/nif.html but it seems like they are > > incredibly unsafe: ?it is also the least safe, because a crash in a NIF > > brings the emulator down too?. > > They're not "incredibly unsafe". There's nothing inherently unstable > about a NIF. It's precisely as stable (or unstable) as the underlying > implementation. It's just that it's running inside the BEAM, which > means that *if* it crashes, it's going to crash the whole Erlang node. > > So: if you think you can write reliable C, go for it. If not, then you > need to look at other options. See Dmytro's email. He skipped "C Node" > as an option, incidentally. Another option is to host the NIF in > another Erlang node, and use Erlang distribution to talk to it. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Wed Aug 16 17:29:36 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Wed, 16 Aug 2017 17:29:36 +0200 Subject: [erlang-questions] OTP 20.0.2 segfault at 5c ip in beam.smp In-Reply-To: References: Message-ID: You can provoke a VM "crash" with erlang:halt(abort) to test if the OS creates a core dump file. /Sverker On 08/16/2017 05:10 PM, Lukas Larsson wrote: > You need to instruct the operating system to save a core dump, > http://man7.org/linux/man-pages/man5/core.5.html. > > Some operating systems can be configured to dump parts of the core dump to > the syslog when a segfault happens, which can be useful but a full core > dump contains much more information so is a lot more useful. > > Lukas > > On Wed, Aug 16, 2017 at 5:00 PM, Gerhard Lazu wrote: > >> The crash dump was never generated because the Erlang VM segfaulted. >> >> Is there a way to force extra debugging output in segfault scenarios? >> >> On Wed, Aug 16, 2017 at 1:42 PM, Lukas Larsson wrote: >> >>> Hello, >>> >>> In order to debug this we need access to a core file generated from the >>> crash and also the beam.smp binary that was used to run it. >>> >>> Lukas >>> >>> On Wed, Aug 16, 2017 at 1:20 PM, Gerhard Lazu wrote: >>> >>>> Hi, >>>> >>>> A RabbitMQ 3.6.11-rc.3 node crashed when OTP 20.0.2 segfaulted. This is >>>> the only information that we have about the Erlang VM itself: >>>> >>>> Aug 16 04:08:59 localhost kernel: [61139.180699] 2_scheduler[15162]: >>>> segfault at 5c ip 00000000005ba760 sp 00007f8fea1ffc40 err >>>> or 4 in beam.smp[400000+333000] >>>> >>>> This is the last line logged by RMQ, 2 seconds before the Erlang VM >>>> segfault: >>>> >>>> =INFO REPORT==== 16-Aug-2017::04:08:57 === closing AMQP connection >>>> <0.24230.9> (10.0.16.35:42562 -> 10.0.16.23:5672, vhost: '/', user: >>>> 'admin') >>>> >>>> beam.smp runs with the following flags: >>>> >>>> /var/vcap/packages/erlang-20.0.2/lib/erlang/erts-9.0.2/bin/beam.smp >>>> -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root >>>> /var/vcap/packages/erlang-20.0.2/lib/erlang -progname erl -- -home >>>> /home/vcap -- -pa /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/ebin >>>> -noshell -noinput -s rabbit boot -sname rabbit@REDACTED -boot >>>> start_sasl -config /var/vcap/jobs/rabbitmq-server/rabbitmq -kernel >>>> inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl >>>> sasl_error_logger false -rabbit error_logger {file,"/var/vcap/sys/log/rabbi >>>> tmq-server/rabbit@REDACTED"} -rabbit sasl_error_logger >>>> {file,"/var/vcap/sys/log/rabbitmq-server/rabbit@REDACTED"} >>>> -rabbit enabled_plugins_file "/var/vcap/jobs/rabbitmq-serve >>>> r/packages/rabbitmq-server/etc/rabbitmq/enabled_plugins" -rabbit >>>> plugins_dir "/var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/plugins" >>>> -rabbit plugins_expand_dir "/var/vcap/store/rabbitmq-serv >>>> er/mnesia/rabbit@REDACTED" -os_mon start_cpu_sup >>>> false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir >>>> "/var/vcap/store/rabbitmq-server/mnesia/rabbit@REDACTED" -kernel >>>> inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 >>>> >>>> We are running on: >>>> >>>> 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC >>>> 2017 x86_64 GNU/Linux >>>> >>>> When this happens again, how do I capture more debugging information? I >>>> would like to file an OTP bug report, but I'm not sure that we have enough >>>> information. >>>> >>>> Apparently, this was reported at least once before, against OTP 18.1: >>>> https://github.com/rabbitmq/rabbitmq-server/issues/459 >>>> >>>> Thank you, Gerhard. >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Wed Aug 16 18:01:11 2017 From: gerhard@REDACTED (Gerhard Lazu) Date: Wed, 16 Aug 2017 17:01:11 +0100 Subject: [erlang-questions] OTP 20.0.2 segfault at 5c ip in beam.smp In-Reply-To: References: Message-ID: Thank you Lukas & Sverker. Patiently waiting for the next Erlang VM segfault : ) On Wed, Aug 16, 2017 at 4:10 PM, Lukas Larsson wrote: > You need to instruct the operating system to save a core dump, > http://man7.org/linux/man-pages/man5/core.5.html. > > Some operating systems can be configured to dump parts of the core dump to > the syslog when a segfault happens, which can be useful but a full core > dump contains much more information so is a lot more useful. > > Lukas > > On Wed, Aug 16, 2017 at 5:00 PM, Gerhard Lazu wrote: > >> The crash dump was never generated because the Erlang VM segfaulted. >> >> Is there a way to force extra debugging output in segfault scenarios? >> >> On Wed, Aug 16, 2017 at 1:42 PM, Lukas Larsson wrote: >> >>> Hello, >>> >>> In order to debug this we need access to a core file generated from the >>> crash and also the beam.smp binary that was used to run it. >>> >>> Lukas >>> >>> On Wed, Aug 16, 2017 at 1:20 PM, Gerhard Lazu >>> wrote: >>> >>>> Hi, >>>> >>>> A RabbitMQ 3.6.11-rc.3 node crashed when OTP 20.0.2 segfaulted. This is >>>> the only information that we have about the Erlang VM itself: >>>> >>>> Aug 16 04:08:59 localhost kernel: [61139.180699] >>>> 2_scheduler[15162]: segfault at 5c ip 00000000005ba760 sp 00007f8fea1ffc40 >>>> err >>>> or 4 in beam.smp[400000+333000] >>>> >>>> This is the last line logged by RMQ, 2 seconds before the Erlang VM >>>> segfault: >>>> >>>> =INFO REPORT==== 16-Aug-2017::04:08:57 === closing AMQP connection >>>> <0.24230.9> (10.0.16.35:42562 -> 10.0.16.23:5672, vhost: '/', user: >>>> 'admin') >>>> >>>> beam.smp runs with the following flags: >>>> >>>> /var/vcap/packages/erlang-20.0.2/lib/erlang/erts-9.0.2/bin/beam.smp >>>> -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root >>>> /var/vcap/packages/erlang-20.0.2/lib/erlang -progname erl -- -home >>>> /home/vcap -- -pa /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/ebin >>>> -noshell -noinput -s rabbit boot -sname rabbit@REDACTED -boot >>>> start_sasl -config /var/vcap/jobs/rabbitmq-server/rabbitmq -kernel >>>> inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl >>>> sasl_error_logger false -rabbit error_logger {file,"/var/vcap/sys/log/rabbi >>>> tmq-server/rabbit@REDACTED"} -rabbit sasl_error_logger >>>> {file,"/var/vcap/sys/log/rabbitmq-server/rabbit@REDACTED"} >>>> -rabbit enabled_plugins_file "/var/vcap/jobs/rabbitmq-serve >>>> r/packages/rabbitmq-server/etc/rabbitmq/enabled_plugins" -rabbit >>>> plugins_dir "/var/vcap/jobs/rabbitmq-serve >>>> r/packages/rabbitmq-server/plugins" -rabbit plugins_expand_dir >>>> "/var/vcap/store/rabbitmq-server/mnesia/rabbit@REDACTED" >>>> -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon >>>> start_memsup false -mnesia dir "/var/vcap/store/rabbitmq-serv >>>> er/mnesia/rabbit@REDACTED" -kernel inet_dist_listen_min 25672 >>>> -kernel inet_dist_listen_max 25672 >>>> >>>> We are running on: >>>> >>>> 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC >>>> 2017 x86_64 GNU/Linux >>>> >>>> When this happens again, how do I capture more debugging information? I >>>> would like to file an OTP bug report, but I'm not sure that we have enough >>>> information. >>>> >>>> Apparently, this was reported at least once before, against OTP 18.1: >>>> https://github.com/rabbitmq/rabbitmq-server/issues/459 >>>> >>>> Thank you, Gerhard. >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Wed Aug 16 18:01:11 2017 From: gerhard@REDACTED (Gerhard Lazu) Date: Wed, 16 Aug 2017 17:01:11 +0100 Subject: [erlang-questions] OTP 20.0.2 segfault at 5c ip in beam.smp In-Reply-To: References: Message-ID: Thank you Lukas & Sverker. Patiently waiting for the next Erlang VM segfault : ) On Wed, Aug 16, 2017 at 4:10 PM, Lukas Larsson wrote: > You need to instruct the operating system to save a core dump, > http://man7.org/linux/man-pages/man5/core.5.html. > > Some operating systems can be configured to dump parts of the core dump to > the syslog when a segfault happens, which can be useful but a full core > dump contains much more information so is a lot more useful. > > Lukas > > On Wed, Aug 16, 2017 at 5:00 PM, Gerhard Lazu wrote: > >> The crash dump was never generated because the Erlang VM segfaulted. >> >> Is there a way to force extra debugging output in segfault scenarios? >> >> On Wed, Aug 16, 2017 at 1:42 PM, Lukas Larsson wrote: >> >>> Hello, >>> >>> In order to debug this we need access to a core file generated from the >>> crash and also the beam.smp binary that was used to run it. >>> >>> Lukas >>> >>> On Wed, Aug 16, 2017 at 1:20 PM, Gerhard Lazu >>> wrote: >>> >>>> Hi, >>>> >>>> A RabbitMQ 3.6.11-rc.3 node crashed when OTP 20.0.2 segfaulted. This is >>>> the only information that we have about the Erlang VM itself: >>>> >>>> Aug 16 04:08:59 localhost kernel: [61139.180699] >>>> 2_scheduler[15162]: segfault at 5c ip 00000000005ba760 sp 00007f8fea1ffc40 >>>> err >>>> or 4 in beam.smp[400000+333000] >>>> >>>> This is the last line logged by RMQ, 2 seconds before the Erlang VM >>>> segfault: >>>> >>>> =INFO REPORT==== 16-Aug-2017::04:08:57 === closing AMQP connection >>>> <0.24230.9> (10.0.16.35:42562 -> 10.0.16.23:5672, vhost: '/', user: >>>> 'admin') >>>> >>>> beam.smp runs with the following flags: >>>> >>>> /var/vcap/packages/erlang-20.0.2/lib/erlang/erts-9.0.2/bin/beam.smp >>>> -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root >>>> /var/vcap/packages/erlang-20.0.2/lib/erlang -progname erl -- -home >>>> /home/vcap -- -pa /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/ebin >>>> -noshell -noinput -s rabbit boot -sname rabbit@REDACTED -boot >>>> start_sasl -config /var/vcap/jobs/rabbitmq-server/rabbitmq -kernel >>>> inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl >>>> sasl_error_logger false -rabbit error_logger {file,"/var/vcap/sys/log/rabbi >>>> tmq-server/rabbit@REDACTED"} -rabbit sasl_error_logger >>>> {file,"/var/vcap/sys/log/rabbitmq-server/rabbit@REDACTED"} >>>> -rabbit enabled_plugins_file "/var/vcap/jobs/rabbitmq-serve >>>> r/packages/rabbitmq-server/etc/rabbitmq/enabled_plugins" -rabbit >>>> plugins_dir "/var/vcap/jobs/rabbitmq-serve >>>> r/packages/rabbitmq-server/plugins" -rabbit plugins_expand_dir >>>> "/var/vcap/store/rabbitmq-server/mnesia/rabbit@REDACTED" >>>> -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon >>>> start_memsup false -mnesia dir "/var/vcap/store/rabbitmq-serv >>>> er/mnesia/rabbit@REDACTED" -kernel inet_dist_listen_min 25672 >>>> -kernel inet_dist_listen_max 25672 >>>> >>>> We are running on: >>>> >>>> 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC >>>> 2017 x86_64 GNU/Linux >>>> >>>> When this happens again, how do I capture more debugging information? I >>>> would like to file an OTP bug report, but I'm not sure that we have enough >>>> information. >>>> >>>> Apparently, this was reported at least once before, against OTP 18.1: >>>> https://github.com/rabbitmq/rabbitmq-server/issues/459 >>>> >>>> Thank you, Gerhard. >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From acautin@REDACTED Thu Aug 17 08:41:14 2017 From: acautin@REDACTED (Agustin Cautin) Date: Thu, 17 Aug 2017 08:41:14 +0200 Subject: [erlang-questions] Erlang 20 Access Violation crash on Windows Message-ID: Hello, We are having crashes in our production systems every 4-6 hours, we don't have much information regarding the crash as the vm just suddenly stops due to access violation the only information we have is from the system event log: Faulting application name: erl.exe, version: 0.0.0.0, time stamp: 0x59497e1a Faulting module name: beam.smp.dll, version: 0.0.0.0, time stamp: 0x59497e05 Exception code: 0xc0000005 Fault offset: 0x0000000000137528 Faulting process id: 0x1720 Faulting application start time: 0x01d311b3967b4ead Faulting application path: d:\K2Informatics\sbsgui\erts-9.0\bin\erl.exe Faulting module path: d:\K2Informatics\sbsgui\erts-9.0\bin\beam.smp.dll Report Id: 069f7e2b-7db4-11e7-acb4-3cd92bfb3e2e This is happening in a cluster of 6 machines but only under load, so far we haven't been able to reproduce the crash on systems outside the production environment, any help in how to get to the root cause or which tools can we use to debug it will be greatly appreciated. Regards. Agustin. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Catenacci@REDACTED Thu Aug 17 14:21:57 2017 From: Catenacci@REDACTED (Onorio Catenacci) Date: Thu, 17 Aug 2017 08:21:57 -0400 Subject: [erlang-questions] Erlang 20 Access Violation crash on Windows Message-ID: Hi Agustin, A few thoughts: 1.) Are you using any NIF code? 2.) Specifically, which OS are you using for your production system and are you using the same OS (down to patches) for the other system that is not crashing? 3.) If I were you I'd be trying to run a memory monitor in the background on the failing machine(s). You might try this: https://docs.microsoft.com/en-us/sysinternals/downloads/rammap 4.) Use this: https://docs.microsoft.com/en-us/sysinternals/downloads/autoruns to see what's automatically getting started on your machine. Shutoff anything that doesn't need to be autostarted. Better yet shut off everything and run it and then slowly add things back till you can repro the issue again. Basically if you're sure that this is only happening on your prod machines and you've got other machines where it's not happening, then figure out what's different. But I can't tell if you've got other machines which aren't seeing the issue and I'd guess you probably don't. -- Onorio Catenacci http://onor.io http://www.google.com/+OnorioCatenacci -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Thu Aug 17 17:48:07 2017 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Thu, 17 Aug 2017 17:48:07 +0200 Subject: [erlang-questions] release_handler taking long till it can handle requests? Message-ID: Hi. We are starting three nodes, connect them, and then run an application on each. We see in the logs SASL entry that says that the child release_handler has been started. If we send any upgrade requests to the system we seem to have to wait about 5s before we it seems to be actually ready. Before we get a noproc exit for any call to release_handler:which_releases(infinity). If we wait these additional 5s, everything goes fine after without issue. If we only wait 3s, release_handler might not have come up on some nodes yet - at least we get this error for an erroneous gen_server call. Applications are started in embedded mode, we're running on full releases here. So, I'm not asking about the performance or timing of the hot code update itself. If we wait the prerequisite time, it just works. No issue there. Given that up to 5s have to be waited even after the logs show the release_handler instances have been started (all nodes on same computer for test purposes, sharing same timebase), how could we determine that the system is ready to handle such requests? We do the request by remote procedure call, so the system itself is not impacted at all. Our upgrades just fail. What's your experience? Is this normal, expected? Thanks and kind regards, Oliver From codewiget95@REDACTED Thu Aug 17 23:46:54 2017 From: codewiget95@REDACTED (code wiget) Date: Thu, 17 Aug 2017 17:46:54 -0400 Subject: [erlang-questions] Erlang VM -s Argument is causing my program to fail Message-ID: Hello, I have read the thread here: https://stackoverflow.com/questions/2080153/erlang-vm-s-argument-misbehaving and have been troubleshooting to no avail. When I run the erlang vm without the -s flag, my function works: bridge_sup:start_link(). Bridge Supervisor Initializing [warning] ClientId is NULL! [warning] ClientId is NULL! Success Success However, if I have the -s flag set, when my function goes on to call another function emqttc:start_link(...) it never returns: Bridge Supervisor Initializing [warning] ClientId is NULL! [warning] ClientId is NULL! I can verify that it is not just a print problem because the program I am connecting to receives no signal. What could possibly be causing this in the Erlang VM? I have also tried using eval to the same effect Thank you in advance! From quantumpotato@REDACTED Fri Aug 18 04:48:34 2017 From: quantumpotato@REDACTED (qp) Date: Thu, 17 Aug 2017 22:48:34 -0400 Subject: [erlang-questions] Games made with Erlworld? Message-ID: I just found https://code.google.com/archive/p/erlworld/ and am pleased to see a working Erlang game engine! I was able to get the Pong example running within a minute. I've tried searching for games made in Erlworld and haven't found any, maybe you know of some? Searching for "Erlang + OpenGL" or "Erlang game" usually ends up on server discussions (like http://www.erlang-factory.com/upload/presentations/395/ErlangandFirst-PersonShooters.pdf ) or pointing to Wings3D the model editor ( http://wings3d.com/ ). The Wings3D forum is taking its time in sending me a confirmation email so I haven't been able to post there yet. When I ran the Erlworld Blastox (Asteroids clone) example, it was tremendously laggy and kept freezing. The other closest thing I've found is ESDL2 which has display but no user input ( https://github.com/ninenines/esdl2/issues/20 ) I'm looking for other examples of Erlang clientside games done with Erlworld as a reference. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Aug 18 11:08:31 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 18 Aug 2017 12:08:31 +0300 Subject: [erlang-questions] lists:mapfind? Message-ID: We have lists:keyfind, but are there any plans for lists:mapfind? -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Fri Aug 18 11:13:12 2017 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 18 Aug 2017 11:13:12 +0200 Subject: [erlang-questions] Proper way to run C code without crashing the VM In-Reply-To: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> References: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Message-ID: Re NIF's safety etc. Here's my take NIFs should be used for one reason only - performance. If performance is not a problem you should use an external port program, or an external c-node. Also the NIF should be called many times per second, not just once now and then - if the NIF is called infrequently then you should use and external port program. The only reason for using a NIF is to eliminate the round-trip times for talking to an external process, and speeding time to convert the internal data structures used by the programs since the NIFs have direct access to the stacks and heaps of the Erlang processes involved. Using a NIF is breaking the guarantee - if you open the package and mess around with the bits inside then don't blame me if you break something ... If you can solve your problem in pure Erlang, then do so and don't use a NIF. There are a large number of reasons not to use NIFs - Your code might break the system in ways you don't understand - Your code will be single-platform and non-portable - Your code will not be 'future proof' - Understanding your code will require knowledge of Erlang + + a build procedure I have always believed that large systems should be made from small communicating components communicating via well-defined protocols - linking things together in memory makes the individual components larger and more unmanageable - we should be trying to do exactly the opposite, making things smaller and isolating them. The other problem with NIFs is that they make alternative implementations pf Erlang far more difficult. An Erlang implementation has four main components: 1) A compiler from Erlang to beam code 2) A beam instruction emulator 3) A load of BIFs (or NIF's if you like) 4) Huge libraries written in Erlang Each of these represent massive effort. Porting Erlang to (say) the JVM or .NET virtual machine would be relatively easy were it not for 3). Changing the compiler to a new instruction set and changing 1) and 2) is not that difficult but any changes the calling sequences would really mess up all the code for the BIFs and NIFs. Never forget that NIFs are an optimization and nothing else (since the same effect can be achieved in an external process) - "Premature Optimization is the root of all evil" [The best form of optimization is patience - in 20 years Erlang has become at least million times faster - of this probably a factor 10 comes from smarter compilation - and 100,000 from faster clock speeds and better hardware. (These are just guess but in 20 years Clock speeds have gone from Mhz to GHz and memories increased from MBytes to GBytes) Pure Erlang code written 30 years ago ofter runs fine and is way faster than it was but the BIFs of 30 years ago are long dead and will not even compile. Patience is a virue] Cheers /Joe On Wed, Aug 16, 2017 at 3:53 PM, code wiget wrote: > Hello, > > I have been reading about NIF?s here: > http://erlang.org/doc/tutorial/nif.html but it seems like they are > incredibly unsafe: ?it is also the least safe, because a crash in a NIF > brings the emulator down too?. The problem here is huge, ideally I can?t > ever have my Erlang VM close. > > What solution do you Erlangers use when you have to run low level code? My > goal is to send a sort of ?GET? request with a UUID and I will receive back > that UUID along with some binary values. My C code works with this, but I > can?t jeopardize the entire server crashing. > > Thank you for your help! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From dmytro.lytovchenko@REDACTED Fri Aug 18 11:13:48 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Fri, 18 Aug 2017 11:13:48 +0200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: Message-ID: A good first step would be to make a ticket on bugs.erlang.org This isn't a primary source of tasks for OTP team but one of the contributors might like it. A good second step would be for someone to write it. 2017-08-18 11:08 GMT+02:00 Max Lapshin : > We have lists:keyfind, but are there any plans for lists:mapfind? > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Aug 18 11:23:35 2017 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 18 Aug 2017 11:23:35 +0200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: Message-ID: I've been thinking of adding that and also lists:mapsort, and while at it we should probably also add lists:mapdelete, lists:mapmap and the others. The reason I haven't spent much effort in adding them is because I'm not sure the lists module is the correct place to put them and also I'm unsure about their general usefulness. Lukas On Fri, Aug 18, 2017 at 11:08 AM, Max Lapshin wrote: > We have lists:keyfind, but are there any plans for lists:mapfind? > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Aug 18 11:23:49 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 18 Aug 2017 12:23:49 +0300 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: Message-ID: Ok. So I wanted to hear something like: a) it is already written, wait a bit b) not written but a good idea: spend some time and write it c) bad idea, will not accept it even if you write it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hachreak@REDACTED Fri Aug 18 11:11:51 2017 From: hachreak@REDACTED (Leo) Date: Fri, 18 Aug 2017 11:11:51 +0200 Subject: [erlang-questions] erlang and elixir protocols Message-ID: Hello everybody, I was wondering if there is any way/project to emulate Elixir Protocols in Erlang? Leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila.r.nohl@REDACTED Fri Aug 18 11:51:25 2017 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Fri, 18 Aug 2017 11:51:25 +0200 Subject: [erlang-questions] Proper way to run C code without crashing the VM In-Reply-To: References: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Message-ID: Hello! I used to use NIFs a couple of times. These were really simple, practically a wrapper around a libc call (e.g. realpath(3)). I think the whole NIF code was less than 100 lines including boilerplate, copyright message comment, etc. and worked on both Linux and Solaris (our target platforms). It seemed easier to setup the build process and bundle the libs with the result than go with an external port. I think it *is* possible to write this small correct C code, so safety wasn't really compromised. On the other hand it was also an opportunity to learn about NIFs :-) 2017-08-18 11:13 GMT+02:00 Joe Armstrong : > Re NIF's safety etc. Here's my take > > NIFs should be used for one reason only - performance. > > If performance is not a problem you should use an external port > program, or an external c-node. > > Also the NIF should be called many times per second, not just once now > and then - if the NIF is called infrequently then you should use and > external port program. > > The only reason for using a NIF is to eliminate the round-trip times > for talking to an external process, and speeding time to convert the > internal data structures used by the programs since the NIFs have > direct access to the stacks and heaps of the Erlang processes > involved. > > Using a NIF is breaking the guarantee - if you open the package and mess > around with the bits inside then don't blame me if you break something > ... > > If you can solve your problem in pure Erlang, then do so and don't use > a NIF. > > There are a large number of reasons not to use NIFs > > - Your code might break the system in ways you don't understand > - Your code will be single-platform and non-portable > - Your code will not be 'future proof' > - Understanding your code will require knowledge of Erlang + > + a build procedure > > I have always believed that large systems should be made from small > communicating components communicating via well-defined protocols - > linking things together in memory makes the individual components > larger and more unmanageable - we should be trying to do exactly the > opposite, making things smaller and isolating them. > > The other problem with NIFs is that they make alternative > implementations pf Erlang far more difficult. An Erlang implementation > has four main components: > > 1) A compiler from Erlang to beam code > 2) A beam instruction emulator > 3) A load of BIFs (or NIF's if you like) > 4) Huge libraries written in Erlang > > Each of these represent massive effort. > > Porting Erlang to (say) the JVM or .NET virtual machine would be > relatively easy were it not for 3). Changing the compiler to a new > instruction set and changing 1) and 2) is not that difficult but any > changes the calling sequences would really mess up all the code for > the BIFs and NIFs. > > Never forget that NIFs are an optimization and nothing else > (since the same effect can be achieved in an external process) - > > "Premature Optimization is the root of all evil" > > [The best form of optimization is patience - in 20 years Erlang has become > at least million times faster - of this probably a factor 10 comes from > smarter compilation - and 100,000 from faster clock speeds and better hardware. > (These are just guess but in 20 years Clock speeds have gone from Mhz to GHz > and memories increased from MBytes to GBytes) > > Pure Erlang code written 30 years ago ofter runs fine and is way faster > than it was but the BIFs of 30 years ago are long dead and will not > even compile. > > Patience is a virue] > > Cheers > > /Joe > > > > On Wed, Aug 16, 2017 at 3:53 PM, code wiget wrote: >> Hello, >> >> I have been reading about NIF?s here: >> http://erlang.org/doc/tutorial/nif.html but it seems like they are >> incredibly unsafe: ?it is also the least safe, because a crash in a NIF >> brings the emulator down too?. The problem here is huge, ideally I can?t >> ever have my Erlang VM close. >> >> What solution do you Erlangers use when you have to run low level code? My >> goal is to send a sort of ?GET? request with a UUID and I will receive back >> that UUID along with some binary values. My C code works with this, but I >> can?t jeopardize the entire server crashing. >> >> Thank you for your help! >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jose.valim@REDACTED Fri Aug 18 12:06:44 2017 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Fri, 18 Aug 2017 10:06:44 +0000 Subject: [erlang-questions] erlang and elixir protocols In-Reply-To: References: Message-ID: It is definitely possible, as Elixir compiles down to Erlang. You can learn more about how protocols work by reading the abstract code (in Erlang) from Elixir .beam files. -- *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Aug 18 12:20:41 2017 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 18 Aug 2017 12:20:41 +0200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: Message-ID: We do like the idea, however some time has to be spent on figuring out what the best API would be. ListOfMaps = [#{ key => 1}, #{ key => 2}], #{ key => 2 } == lists:keyfind(fun(#{ key := V }) -> V end, 2, ListOfMaps). #{ key => 2 } == lists:keyfind(key, 2, ListOfMaps). #{ key => 2 } == lists:find(fun(#{ key := V }) -> V end, 2, ListOfMaps). #{ key => 2 } == lists:mapfind(key, 2, ListOfMaps). #{ key => 2 } == maps:findinlist(key, 2, ListOfMaps). Maybe there are other, better ideas? My preferences are either lists:keyfind with a fun, or lists:mapfind. I like lists:keyfind with a fun as it is more general, and allows me to use any container within a list, it is however not as clear is lists:mapfind in what the function does. Also as I stated before, if we add calls for find, we should also add calls for the other variations of the function. On Fri, Aug 18, 2017 at 11:23 AM, Max Lapshin wrote: > Ok. > > So I wanted to hear something like: > > a) it is already written, wait a bit > b) not written but a good idea: spend some time and write it > c) bad idea, will not accept it even if you write it. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hawk.mattsson@REDACTED Fri Aug 18 13:40:03 2017 From: hawk.mattsson@REDACTED (=?UTF-8?Q?H=C3=A5kan_Mattsson?=) Date: Fri, 18 Aug 2017 13:40:03 +0200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: Message-ID: Are really the lists module the natural home for these functions? Would not the maps module be a better choice? /H?kan On Fri, Aug 18, 2017 at 11:23 AM, Lukas Larsson wrote: > I've been thinking of adding that and also lists:mapsort, and while at it > we should probably also add lists:mapdelete, lists:mapmap and the others. > > The reason I haven't spent much effort in adding them is because I'm not > sure the lists module is the correct place to put them and also I'm unsure > about their general usefulness. > > Lukas > > On Fri, Aug 18, 2017 at 11:08 AM, Max Lapshin > wrote: > >> We have lists:keyfind, but are there any plans for lists:mapfind? >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Fri Aug 18 14:58:01 2017 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 18 Aug 2017 13:58:01 +0100 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: Message-ID: On 18 August 2017 at 11:20, Lukas Larsson wrote: > We do like the idea, however some time has to be spent on figuring out what > the best API would be. What's wrong with plain ol' lists:filter/2? From zxq9@REDACTED Fri Aug 18 15:04:52 2017 From: zxq9@REDACTED (zxq9) Date: Fri, 18 Aug 2017 22:04:52 +0900 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: Message-ID: <3705424.6ZUeSbM6V7@changa> On 2017?08?18? ??? 13:58:01 Roger Lipscombe wrote: > On 18 August 2017 at 11:20, Lukas Larsson wrote: > > We do like the idea, however some time has to be spent on figuring out what > > the best API would be. > > What's wrong with plain ol' lists:filter/2? Good point. What are the cases not covered by the lists module? filter/2, partition/2, matching/guarded list comprehensions, etc? I'm not trying to say there are no such cases or that an improvement can't be made -- I just can't think of a situation off the top of my head that would be served by something like lists:mapfind/3,4. -Craig From djisthegreat@REDACTED Fri Aug 18 16:30:24 2017 From: djisthegreat@REDACTED (Sachin Joshi) Date: Fri, 18 Aug 2017 20:00:24 +0530 Subject: [erlang-questions] Differentiating between FQDN and IP addresses Message-ID: HI Team, I am getting an input in a program as 'ip address' or 'FQDN'. I need to identify if it is an ipv4 or ipv6 or a hostname(fqdn). Can someone help me in this? Thanks, Sachin -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Fri Aug 18 17:00:38 2017 From: zxq9@REDACTED (zxq9) Date: Sat, 19 Aug 2017 00:00:38 +0900 Subject: [erlang-questions] Differentiating between FQDN and IP addresses In-Reply-To: References: Message-ID: <12960789.MTf3YIpLEP@changa> On 2017?08?18? ??? 20:00:24 Sachin Joshi wrote: > I am getting an input in a program as 'ip address' or 'FQDN'. I need to > identify if it is an ipv4 or ipv6 or a hostname(fqdn). > Can someone help me in this? Hi, Sachin. I'm assuming you are receiving these as strings. The function inet:parse_strict_address/1 can help you determine if the string is a valid IPv4, IPv6, or invalid IP address. http://erlang.org/doc/man/inet.html#parse_strict_address-1 If it is an invalid IP address, then attempt to use it as a FQDN instead. If that fails, then perhaps it is invalid in general. As for "what is a valid FQDN?" the standard has slid a lot in recent years, so it seems more common to just try it out (perhaps converting a utf-8 string via punycode first to see if that fails) and see if it barfs or not. Imagine something along the lines of... > -spec interpret_address(string()) -> Result > when Result :: {ipv4, inet:ipv4_address()} > | {ipv6, inet:ipv6_address()} > | {fqdn, inet:hostname()}. > > interpret_address(String) -> > case inet:parse_strict_address(String) of > {A, B, C, D} -> > {ipv4, {A, B, C, D}}; > {A, B, C, D, E, F, G, H} -> > {ipv6, {A, B, C, D, E, F, G, H}}; > {error, einval} -> > {fqdn, String} > end. With the exception that the result on {error, einval} should really look like: > {error, einval} -> > validate_fqdn(String) and the overall function should be allowed to return the type > Result :: {ipv4, inet:ipv4_address()} > | {ipv6, inet:ipv6_address()} > | {fqdn, inet:hostname()} > | {error, bad_address}. But you can get the effect of this by just trying the return value of {fqdn, String} and see if it can actually resolve a hostname this way. Or something like that. Forgive my imprecision -- it is very late here. Hopefully this points you in the right direction. -Craig From vincent.dephily@REDACTED Fri Aug 18 18:08:58 2017 From: vincent.dephily@REDACTED (Vincent de Phily) Date: Fri, 18 Aug 2017 18:08:58 +0200 Subject: [erlang-questions] Proper way to run C code without crashing the VM In-Reply-To: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> References: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Message-ID: On 2017-08-16 15:53, code wiget wrote: > Hello, > > I have been reading about NIF?s > here: http://erlang.org/doc/tutorial/nif.html but it seems like they are > incredibly unsafe: ?it is also the least safe, because a crash in a NIF > brings the emulator down too?. The problem here is huge, ideally I can?t > ever have my Erlang VM close. Whatever you do, your node *will* close someday, if only due to hardware failure. If you have high uptime requirements, you should tackle node redundancy/failover first before worrying about individual node stability. > What solution do you Erlangers use when you have to run low level code? The general advice with NIFs is to keep them small/simple/auditable enough that you can consider them bug-free. Most NIFs are just wrappers around existing libs, so they're going to be as stable as the underlying lib. If the underlying lib is unstable, interface with it using an Erlang port or via the network. If you're writing a complex lib yourself (so using NIFs for performance's sake rather than code reuse), maybe write it in Rust rather than C/C++ ? I'm looking for an excuse to do that myself. See https://www.youtube.com/watch?v=lSLTwWqTbKQ http://hansihe.com/2017/02/05/rustler-safe-erlang-elixir-nifs-in-rust.html https://github.com/hansihe/Rustler > My goal is to send a sort of ?GET? request with a UUID and I will > receive back that UUID along with some binary values. My C code works > with this, but I can?t jeopardize the entire server crashing. Sounds like you're querying an external server, not doing a lot of computation locally ? If that's the case, going low-level with NIFs, ports, etc is probably overkill, you're better off implementing it in pure Erlang. Actually, even if you do heavy computation locally, try implementing it in pure Erlang first : chances are that performance will be good enough, and you'll have an easyer time debugging and tuning Erlang than C. -- Vincent de Phily From marco.molteni@REDACTED Sat Aug 19 12:07:42 2017 From: marco.molteni@REDACTED (Marco Molteni) Date: Sat, 19 Aug 2017 12:07:42 +0200 Subject: [erlang-questions] Games made with Erlworld? In-Reply-To: References: Message-ID: I stumbled upon Erlworld some time ago and was excited as you are. Then I kicked the tires and the thing is unusable in my opinion: it is very laggy and if you keep it running WxWidgets starts leaking like mad and brings down the Erlang VM. (WxWidgets as used by Erlworld, not WxWidgets by itself). Also if you have a look at the code, it is an attempt of doing OOP in Erlang, which clearly is not a good fit. Too bad :-/ marco > On 18 Aug 2017, at 04:48, qp wrote: > > I just found https://code.google.com/archive/p/erlworld/ and am pleased to see a working Erlang game engine! I was able to get the Pong example running within a minute. > > I've tried searching for games made in Erlworld and haven't found any, maybe you know of some? > > Searching for "Erlang + OpenGL" or "Erlang game" usually ends up on server discussions (like http://www.erlang-factory.com/upload/presentations/395/ErlangandFirst-PersonShooters.pdf ) > or pointing to Wings3D the model editor ( http://wings3d.com/ ). > > The Wings3D forum is taking its time in sending me a confirmation email so I haven't been able to post there yet. > > When I ran the Erlworld Blastox (Asteroids clone) example, it was tremendously laggy and kept freezing. > > The other closest thing I've found is ESDL2 which has display but no user input ( https://github.com/ninenines/esdl2/issues/20 ) > > I'm looking for other examples of Erlang clientside games done with Erlworld as a reference. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Sat Aug 19 17:05:12 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 19 Aug 2017 15:05:12 +0000 Subject: [erlang-questions] Proper way to run C code without crashing the VM In-Reply-To: References: <1AE39D60-26F6-4CED-B40C-E2C295DD3171@gmail.com> Message-ID: I did an experiment a couple of years ago: 1. Run an OCaml program behind a port from Erlang 2. *Pipeline* requests to the port. That is, don't wait for the roundtrip of each message, just shove as many requests you can to the port and have it handle them. The OCaml program just reflects whatever it got back (an integer counter). 3. Use one CPU core for Erlang, once for OCaml. This ran about 1 million req/s on a typical modern laptop. That is, the amortized message overhead is about 1us. It suggests that the NIF usecase is that you have many small calls which are latency bound (my above test is a throughput one due to the amortization). If you have large datastructures there may also be a reason for a NIF in some cases. Measurement is key. I think there are many cases where you don't really need a NIF and a port would do fine. On Fri, Aug 18, 2017 at 11:13 AM Joe Armstrong wrote: > Re NIF's safety etc. Here's my take > > NIFs should be used for one reason only - performance. > > If performance is not a problem you should use an external port > program, or an external c-node. > > Also the NIF should be called many times per second, not just once now > and then - if the NIF is called infrequently then you should use and > external port program. > > The only reason for using a NIF is to eliminate the round-trip times > for talking to an external process, and speeding time to convert the > internal data structures used by the programs since the NIFs have > direct access to the stacks and heaps of the Erlang processes > involved. > > Using a NIF is breaking the guarantee - if you open the package and mess > around with the bits inside then don't blame me if you break something > ... > > If you can solve your problem in pure Erlang, then do so and don't use > a NIF. > > There are a large number of reasons not to use NIFs > > - Your code might break the system in ways you don't understand > - Your code will be single-platform and non-portable > - Your code will not be 'future proof' > - Understanding your code will require knowledge of Erlang + > + a build procedure > > I have always believed that large systems should be made from small > communicating components communicating via well-defined protocols - > linking things together in memory makes the individual components > larger and more unmanageable - we should be trying to do exactly the > opposite, making things smaller and isolating them. > > The other problem with NIFs is that they make alternative > implementations pf Erlang far more difficult. An Erlang implementation > has four main components: > > 1) A compiler from Erlang to beam code > 2) A beam instruction emulator > 3) A load of BIFs (or NIF's if you like) > 4) Huge libraries written in Erlang > > Each of these represent massive effort. > > Porting Erlang to (say) the JVM or .NET virtual machine would be > relatively easy were it not for 3). Changing the compiler to a new > instruction set and changing 1) and 2) is not that difficult but any > changes the calling sequences would really mess up all the code for > the BIFs and NIFs. > > Never forget that NIFs are an optimization and nothing else > (since the same effect can be achieved in an external process) - > > "Premature Optimization is the root of all evil" > > [The best form of optimization is patience - in 20 years Erlang has become > at least million times faster - of this probably a factor 10 comes from > smarter compilation - and 100,000 from faster clock speeds and better > hardware. > (These are just guess but in 20 years Clock speeds have gone from Mhz to > GHz > and memories increased from MBytes to GBytes) > > Pure Erlang code written 30 years ago ofter runs fine and is way faster > than it was but the BIFs of 30 years ago are long dead and will not > even compile. > > Patience is a virue] > > Cheers > > /Joe > > > > On Wed, Aug 16, 2017 at 3:53 PM, code wiget wrote: > > Hello, > > > > I have been reading about NIF?s here: > > http://erlang.org/doc/tutorial/nif.html but it seems like they are > > incredibly unsafe: ?it is also the least safe, because a crash in a NIF > > brings the emulator down too?. The problem here is huge, ideally I can?t > > ever have my Erlang VM close. > > > > What solution do you Erlangers use when you have to run low level code? > My > > goal is to send a sort of ?GET? request with a UUID and I will receive > back > > that UUID along with some binary values. My C code works with this, but I > > can?t jeopardize the entire server crashing. > > > > Thank you for your help! > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Aug 21 00:46:37 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 21 Aug 2017 10:46:37 +1200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: Message-ID: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> On 18/08/17 9:23 PM, Lukas Larsson wrote: > I've been thinking of adding that and also lists:mapsort, and while at > it we should probably also add lists:mapdelete, lists:mapmap and the others. > > The reason I haven't spent much effort in adding them is because I'm not > sure the lists module is the correct place to put them and also I'm > unsure about their general usefulness. Three observations: (1) lists: is absolutely the wrong place for them. They belong in the maps: module, where you can find maps:filter(Pred, Map1) -> Map2 >>>> maps:find(Key, Map) -> {ok,Value} | error maps:fold(Fun, Init, Map) -> Acc ... Anybody who wants these functions is most certainly going to look for them there first. Putting the functions in maps: means you don't have to stick 'map' in their names. (2) The names give me only the vaguest idea of what the functions are supposed to do. For example, how is mapdelete different from maps:take/2? lists:delete/2 removes the first copy of its first argument; for maps, does this look for a value, a key, or a pair? How would lists:mapfind have differed from maps:find/2? (3) Given that maps:to_list(Map) -> [{Key,Value}] and maps:from_list(List_Of_Pairs) -> Map exist, additional functions on maps can be prototyped by anyone who wants them in plain Erlang. Presumably map(Fun, Map) -> maps:from_list([{K,Fun(K,V)} || {K,V} <- maps:to_list(Map)]). Let's try three variants of delete/2. keydelete(Key, Map) -> case maps:take(Key, Map) of {_,Map1} -> Map1 ; error -> Map end. valuedelete(Value, Map) -> Sorry, the definition just fell apart. In a map, which is the "first" pair with a matching value? Something that deletes *all* pairs with matching values, no problem. Just use filter. But the first? pairdelete(Key, Value, Map) -> maps:filter(fun (K, V) -> K == Key and V == Value end, Map). It would be a very good thing when proposing a new library function to provide - a specification of what it should do - ideally, a reference implementation in Erlang - a use case. Sometimes, of course, it would be very difficult to write the reference implementation, and sometimes the point of proposing a new function is to discuss what its specification should be. Ah. It just occurred to me that maybe lists:mapfind was meant to be lists:mapfind/3 with an interface like mapfind(Value, Key, MapList) -> the first Map in MapList having a pair Key := Value or false if there is no such Map. % in maps:, so no maps: prefixes. find_in_list(Value, Key, [Map|MapList]) -> case find(Key, Map) of {ok, Value} -> Map ; error -> find_in_list(Value, Key, MapList) end; find_in_list(_, _, []) -> false. But I am left wondering why this particular combination is so important. Suppose instead we add this function to lists:. % first(Pred, List) -> {ok, Value} | error % Types % Value = T % List = [T] % Pred = fun(T -> boolean()) % % If there is a Value in List such that Pred(Value), % returns {ok,Value} for the first such Value, % otherwise returns error. first(Pred, [X|Xs]) -> case Pred(X) of true -> {ok,X} ; false -> first(Pred, Xs) end; first(_, []) -> error. Now find_in_list(Value, Key, MapList) -> lists:first(fun (Map) -> maps:is_key(Key, Map) andalso maps:get(Key, Map) == Value end, MapList). And now we see the primitive we really want: % has_key(Key, Map, Value) -> boolean() % Types % Key = term() % Map = #{} % Value = term() % Returns true when Map contains Key as a key and associates % Value with that key, otherwise false. % The call fails with a {badmap,Map} exception if Map is not a map. maps:has(Key, Map, Value) -> is_key(Key, Map) andalso get(Key, Map) == Value. With that, find_in_list(Value, Key, MapList) -> lists:first(fun(Map) -> maps:has(Key, Map, Value) end, MapList). I have found (the equivalent of) maps:has_key/3 very useful when processing SGML and XML. I have found (the equivalent of) lists:first/2 so useful elsewhere that I was surprised to find it missing from lists:. From ok@REDACTED Mon Aug 21 01:22:10 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 21 Aug 2017 11:22:10 +1200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: <3705424.6ZUeSbM6V7@changa> References: <3705424.6ZUeSbM6V7@changa> Message-ID: On 19/08/17 1:04 AM, zxq9 wrote: > Good point. > > What are the cases not covered by the lists module? > filter/2, partition/2, matching/guarded list comprehensions, etc? We have all(Pred, List) any(Pred, List) but one(Pred, List) is missing. % one(Pred, List) is true when List has exactly one element X % such that Pred(X) is true. Yes, this is useful. one(Pred, [X|Xs]) -> case Pred(X) of true -> not(any(Pred, Xs)) ; false -> one(Pred, Xs) end; one(_, []) -> false. % first(Pred, List) returns {ok,Value} when Value is the % first element of List such that Pred(Value), or error % if there is no such value. first(Pred, [X|Xs]) -> case Pred(X) of true -> {ok,X} ; false -> first(Pred, Xs) end; first(_, []) -> error. % last(Pred, List) returns {ok,Value} when Value is the % last element of List such that Pred(Value), or error % if there is no such value. last(Pred, [X|Xs]) -> case Pred(X) of true -> last_aux(Pred, Xs, X) ; false -> last(Pred, Xs) end; last(_, []) -> error. last_aux(Pred, [X|Xs], Y) -> case Pred(X) of true -> last_aux(Pred, Xs, X) ; false -> last_aux(Pred, Xs, Y) end; last_aux(_, _, Y) -> {ok, Y}. % foldl1(Fun, List) is a relative of foldl/3 used when % the function has no (representable) left identity, % such as max and min. It is an error if List is empty. foldl1(Fun, [X|Xs]) -> foldl(Fun, X, Xs). foldr1 is also missing. The whole scan{l,r}{,1} family are missing. As mentioned several years ago, unfold/2 is missing. unfold(Fun, State) -> case Fun(State) of {Head,State1} -> [Head | unfold(Fun, State1)] ; [] -> [] end. takewhile and dropwhile are there, and drop is there under the name nthtail, but take is not there. uniq(List) -> List' uniq(Eql, List) -> List' returns a subsequence of List with *adjacent* duplicates dropped, Whenever there is a run ...x1,x2,...,xn such that Eql(xk_1, xk) for 2<=k<=n, x1 is included in the result and x2...xn are not. The default equality function is exact equality. O(length(List)). nub(List) -> List' nub(Eql, List) -> List' returns a subsequence of List with duplicates dropped. Whenever there is a pair ...xi...xj... such that Eql(xi, xj), xj is excluded from the result. The default equality function is exact equality. O(length(List)**2). group(List) -> List_Of_Lists group(Eql, List) -> List_Of_Lists returns a list of lists whose concatenation is the original List. If [...xi...xj...] is an element of the result, Eql(xi, xj). The Eql function should be transitive but need not be an equivalence; the default is exact equality. O(length(List)). From max.lapshin@REDACTED Mon Aug 21 04:13:17 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 21 Aug 2017 05:13:17 +0300 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: <3705424.6ZUeSbM6V7@changa> Message-ID: maps:listfind maps:listtake maps:listpartition so? -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Mon Aug 21 09:30:12 2017 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 21 Aug 2017 08:30:12 +0100 Subject: [erlang-questions] lists:mapfind? In-Reply-To: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> Message-ID: On 20 August 2017 at 23:46, Richard A. O'Keefe wrote: > (1) lists: is absolutely the wrong place for them. The thing I missed at first is this: In Lukas's suggestion, these are *list of maps*, and we're searching the *list* for the *map* with the specified K/V pair. If we're dealing with *lists* (of maps), then the 'lists' module is, at first glance, a reasonable place to put this functionality. It's unclear whether that's what Max was asking for. Max? But: I don't believe we need it *at all*. Let's just generalise the whole thing with a predicate, and call it done. I'd like to see some of ROK's suggestions ('one', 'first', etc.) -- because I came from C#, which has 'Single', 'First', etc., and I kinda miss them. Cheers, Roger. From dmkolesnikov@REDACTED Mon Aug 21 09:33:22 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Mon, 21 Aug 2017 10:33:22 +0300 Subject: [erlang-questions] LiDAR and Erlang Message-ID: <3C18F6BA-4F74-4B10-ADAE-F51A501B80AC@gmail.com> Hello Community, Have you heard about open source utilities, library written on Erlang to deal with LiDAR data, point cloud, etc? I am trying to google the answer on the question as well but input from first hand experience is always appreciated. Best Regards, Dmitry From povetkin.k@REDACTED Mon Aug 21 09:38:27 2017 From: povetkin.k@REDACTED (=?utf-8?B?0JrQvtC90YHRgtCw0L3RgtC40L0g0J/QvtCy0LXRgtC60LjQvQ==?=) Date: Mon, 21 Aug 2017 10:38:27 +0300 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> Message-ID: <820DD611-8E95-4372-B3DD-AF186FAC14F5@gmail.com> > 21 ???. 2017 ?., ? 10:30, Roger Lipscombe ???????(?): > > because I came from C#, > which has 'Single', 'First', etc., and I kinda miss them. In case of ?Single? and ?First? you can use erlang:hd() or [FirstOrSingle | _Tail]. If you can provide an example you are getting challenges with, it?ll be easier to suggest you the most suitable erlang way to solve it. In my experiance: lists:fold[l/r], lists:map, lists:foreach are able to do any kind of jobs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc@REDACTED Mon Aug 21 11:57:15 2017 From: marc@REDACTED (Marc Worrell) Date: Mon, 21 Aug 2017 11:57:15 +0200 Subject: [erlang-questions] Adding file and line information to erl_syntax trees. Message-ID: <44632580-A263-47DD-9641-4908D18A3279@worrell.nl> Hi, In Zotonic (the Erlang CMS) we are using template_compiler [1] to compile ErlyDTL templates to beam modules. I would like to include filename and line positions from the original template files in the generated beam code. This to ease debugging, especially in the case of crashes. I am a bit at a loss how to best include this information into erl_syntax abstract trees. Does anybody have an example how to include this information? Thanks, Marc [1] https://github.com/zotonic/template_compiler From z@REDACTED Mon Aug 21 13:36:05 2017 From: z@REDACTED (Danil Zagoskin) Date: Mon, 21 Aug 2017 14:36:05 +0300 Subject: [erlang-questions] Memory leak in SSL In-Reply-To: References: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> Message-ID: Hi Ingela! We've hit presumably the same bug on OTP 20. That's what I found: 1. This patch https://github.com/erlang/otp/commit/256e01ce80b3aadd63f303b9bda5722ad313220f -- we start invalidation only on Size == Max 2. (flussonic@REDACTED)33> (sys:get_state(ssl_manager))#state.session_cache_server_max. > 1000 3. (flussonic@REDACTED)37> redbug:start("ssl_session_cache:size/1 -> return", [{msgs, 2}]). ... ssl_session_cache:size/1 -> 8610379 Seems like on high load if we accept some new sessions while session_validation does its job, cache size limiting stops working. On Tue, Aug 8, 2017 at 4:55 PM, Ingela Andin wrote: > Hi! > > Depending on the version of the ssl application you may also set these > values. I guess you have an older version as they default to 1000. > > From doc: > > *"session_cache_client_max = integer() * > > Limits the growth of the clients session cache, that is how many sessions > towards servers that are cached to be used by new client connections. If > the maximum number of sessions is reached, the current cache entries will > be invalidated regardless of their remaining lifetime. Defaults to 1000. > * session_cache_server_max = integer() * > > Limits the growth of the servers session cache, that is how many client > sessions are cached by the server. If the maximum number of sessions is > reached, the current cache entries will be invalidated regardless of their > remaining lifetime. Defaults to 1000." > > > Regards Ingela Erlang/OTP Team - Ericsson AB > > > > > 2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov : > >> Hello, >> >> Have you tried to disable reuse of ssl sessions? >> {ssl_options, [{reuse_sessions, false}]} >> >> And reduce the session time >> -ssl session_lifetime 120 >> >> Best Regards, >> Dmitry >> >> > On 8 Aug 2017, at 11.59, Max Treskin wrote: >> > >> > Hello, >> > >> > I have two different HTTPS-servers (different purposes, code, etc) >> built on top of cowboy/ranch, and both have memory leaks. >> > ETS table server_ssl_otp_session_cache has millions of records just >> after hours of work and consumes gigabytes of RAM. >> > What should I do to eliminate this? Is it bug or intended behaviour? >> > >> > Thanks >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Mon Aug 21 14:11:57 2017 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 21 Aug 2017 13:11:57 +0100 Subject: [erlang-questions] lists:mapfind? In-Reply-To: <820DD611-8E95-4372-B3DD-AF186FAC14F5@gmail.com> References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> <820DD611-8E95-4372-B3DD-AF186FAC14F5@gmail.com> Message-ID: On 21 August 2017 at 08:38, ?????????? ???????? wrote: > In case of ?Single? and ?First? you can use erlang:hd() or [FirstOrSingle | > _Tail]. I was referring to Single and First with predicates, which aren't quite the same (though can be built with) hd or [H | _]. My point is that I don't see the need for lists:mapfind (or whatever). Is it a performance thing? From ingela.andin@REDACTED Mon Aug 21 15:23:58 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 21 Aug 2017 15:23:58 +0200 Subject: [erlang-questions] Memory leak in SSL In-Reply-To: References: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> Message-ID: Hi! 2017-08-21 13:36 GMT+02:00 Danil Zagoskin : > Hi Ingela! > > We've hit presumably the same bug on OTP 20. > That's what I found: > 1. This patch https://github.com/erlang/otp/commit/ > 256e01ce80b3aadd63f303b9bda5722ad313220f > -- we start invalidation only on Size == Max > 2. (flussonic@REDACTED)33> (sys:get_state(ssl_manager))# > state.session_cache_server_max. > > 1000 > 3. (flussonic@REDACTED)37> redbug:start("ssl_session_cache:size/1 -> > return", [{msgs, 2}]). > ... > ssl_session_cache:size/1 -> 8610379 > > Seems like on high load if we accept some new sessions while > session_validation does its job, cache size limiting stops working. > > Humm... maybe something was changed, that changed the criteria breaking the assumption, let me ponder that some more. Would it work for you if the test was changed to Size >= Max ? Regards Ingela Erlang/OTP Team - Ericsson AB > On Tue, Aug 8, 2017 at 4:55 PM, Ingela Andin > wrote: > >> Hi! >> >> Depending on the version of the ssl application you may also set these >> values. I guess you have an older version as they default to 1000. >> >> From doc: >> >> *"session_cache_client_max = integer() * >> >> Limits the growth of the clients session cache, that is how many sessions >> towards servers that are cached to be used by new client connections. If >> the maximum number of sessions is reached, the current cache entries will >> be invalidated regardless of their remaining lifetime. Defaults to 1000. >> * session_cache_server_max = integer() * >> >> Limits the growth of the servers session cache, that is how many client >> sessions are cached by the server. If the maximum number of sessions is >> reached, the current cache entries will be invalidated regardless of their >> remaining lifetime. Defaults to 1000." >> >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> >> >> >> 2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov : >> >>> Hello, >>> >>> Have you tried to disable reuse of ssl sessions? >>> {ssl_options, [{reuse_sessions, false}]} >>> >>> And reduce the session time >>> -ssl session_lifetime 120 >>> >>> Best Regards, >>> Dmitry >>> >>> > On 8 Aug 2017, at 11.59, Max Treskin wrote: >>> > >>> > Hello, >>> > >>> > I have two different HTTPS-servers (different purposes, code, etc) >>> built on top of cowboy/ranch, and both have memory leaks. >>> > ETS table server_ssl_otp_session_cache has millions of records just >>> after hours of work and consumes gigabytes of RAM. >>> > What should I do to eliminate this? Is it bug or intended behaviour? >>> > >>> > Thanks >>> > >>> > >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > Danil Zagoskin | z@REDACTED > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Aug 21 18:18:56 2017 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 21 Aug 2017 18:18:56 +0200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> <820DD611-8E95-4372-B3DD-AF186FAC14F5@gmail.com> Message-ID: I agree with Richard that lists:first/2[1] is missing. Adding that functions could solve part of the problem. maps:has/3 could also be useful, though so far I've always found the same functionality to be easier to do with syntax. For me, adding lists:mapfind/3 and friends to the stdlib is a convenience so that I don't end up writing the following in my projects: %% Map helpers mapsort(Key, Maps) -> lists:sort(fun(#{ Key := V1 }, #{ Key := V2 }) -> V1 < V2 end, Maps). mapfind(Key, Value, List) -> lists:find(fun(#{ Key := V } ) -> V == Value end, List). If it is only me and Max that find that we keep re-implementing these functions in various places, then I don't think that we should add special functions for list of maps. However as maps are gradually replacing more and more use-cases where before we used proplists or records, I think these functions will be become increasingly useful. I could very well be wrong though. As for the naming of the functions, I don't have any great suggestions about what they should be called or where they should be placed. I don't find the lists module to be a very wrong place to put them (for the reason Roger pointed out), the maps module also makes some sense, or possibly adding a new module all together. [1]: though I think it should be called lists:find/2 for consistency with lists:keyfind/3 and to not be confused as the opposite of lists:last/1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Aug 22 03:13:12 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 22 Aug 2017 13:13:12 +1200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> Message-ID: On 21/08/17 7:30 PM, Roger Lipscombe wrote: > On 20 August 2017 at 23:46, Richard A. O'Keefe wrote: >> (1) lists: is absolutely the wrong place for them. > > The thing I missed at first is this: > > In Lukas's suggestion, these are *list of maps*, and we're searching > the *list* for the *map* with the specified K/V pair. If we're dealing > with *lists* (of maps), then the 'lists' module is, at first glance, a > reasonable place to put this functionality. The problem is that lists are such a general-purpose data structure. Suppose we had an analogue of select(2) that took a list of io devices and returned one of them that was ready, would we put that in the lists: module or would we expect it to be in io:? For me, the important thing is not whether the elements are held in a list or a tuple or a set or whatever but what are the focal things we're acting on. Of course, this is arguably an instance of the expression problem, and such there may be no best answer. From ok@REDACTED Tue Aug 22 03:15:37 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 22 Aug 2017 13:15:37 +1200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: <820DD611-8E95-4372-B3DD-AF186FAC14F5@gmail.com> References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> <820DD611-8E95-4372-B3DD-AF186FAC14F5@gmail.com> Message-ID: <6861c5bb-d613-5694-2414-37b5bd87a8b4@cs.otago.ac.nz> On 21/08/17 7:38 PM, ?????????? ???????? wrote: > >> 21 ???. 2017 ?., ? 10:30, Roger Lipscombe > > ???????(?): >> >> because I came from C#, >> which has 'Single', 'First', etc., and I kinda miss them. > > In case of ?Single? and ?First? you can use erlang:hd() or > [FirstOrSingle | _Tail]. That doesn't do one/2. And it does first/2 at the expense of extra work, by a possibly very large ratio. > If you can provide an example you are getting challenges with, it?ll be > easier to suggest you the most suitable erlang way to solve it. > In my experiance: lists:fold[l/r], lists:map, lists:foreach are able to > do any kind of jobs. Those functions cannot stop early. all/2 and any/2 can, of course, but they cannot return a non-Boolean result. From erlang.org@REDACTED Tue Aug 22 03:23:47 2017 From: erlang.org@REDACTED (Stanislaw Klekot) Date: Tue, 22 Aug 2017 03:23:47 +0200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> Message-ID: <20170822012347.GA28080@jarowit.net> On Tue, Aug 22, 2017 at 01:13:12PM +1200, Richard A. O'Keefe wrote: > On 21/08/17 7:30 PM, Roger Lipscombe wrote: > >On 20 August 2017 at 23:46, Richard A. O'Keefe wrote: > >> (1) lists: is absolutely the wrong place for them. > > > >The thing I missed at first is this: > > > >In Lukas's suggestion, these are *list of maps*, and we're searching > >the *list* for the *map* with the specified K/V pair. If we're dealing > >with *lists* (of maps), then the 'lists' module is, at first glance, a > >reasonable place to put this functionality. > > The problem is that lists are such a general-purpose data structure. > Suppose we had an analogue of select(2) that took a list of io devices > and returned one of them that was ready, would we put that in the > lists: module or would we expect it to be in io:? This one is easy: `io' module, because the primary focus is not data structure manipulation, but reading or writing. Actually, I initially thought the mapfind() function to be more like map() + find() (a.k.a. filter()). It's not uncommon for me to want to combine map() and some other operation. I treat lists:keyfind() as a function that would normally belonged to proplists module, except that it's mainly intended for 3+ tuples. [select(2) analogue that works with lists] > For me, the important thing is not whether the elements are > held in a list or a tuple or a set or whatever but what are > the focal things we're acting on. > > Of course, this is arguably an instance of the expression problem, > and such there may be no best answer. Well, at least two people agree on what to expect in this case. -- Stanislaw Klekot From zxq9@REDACTED Tue Aug 22 03:37:07 2017 From: zxq9@REDACTED (zxq9) Date: Tue, 22 Aug 2017 10:37:07 +0900 Subject: [erlang-questions] LiDAR and Erlang In-Reply-To: <3C18F6BA-4F74-4B10-ADAE-F51A501B80AC@gmail.com> References: <3C18F6BA-4F74-4B10-ADAE-F51A501B80AC@gmail.com> Message-ID: <2443831.hCvMfGToGR@changa> On 2017?08?21? ??? 10:33:22 Dmitry Kolesnikov wrote: > Hello Community, > > Have you heard about open source utilities, library written on Erlang to deal with LiDAR data, point cloud, etc? No, sadly I haven't. I've got some contact with the LiDAR and gis community and would love to write a library of utilities and interfaces related to this. But I imagine a project like that would never get funded. :-/ -Craig From mjtruog@REDACTED Tue Aug 22 03:46:12 2017 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 21 Aug 2017 18:46:12 -0700 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> Message-ID: <599B8CE4.8020809@gmail.com> On 08/21/2017 06:13 PM, Richard A. O'Keefe wrote: > > > On 21/08/17 7:30 PM, Roger Lipscombe wrote: >> On 20 August 2017 at 23:46, Richard A. O'Keefe wrote: >>> (1) lists: is absolutely the wrong place for them. >> >> The thing I missed at first is this: >> >> In Lukas's suggestion, these are *list of maps*, and we're searching >> the *list* for the *map* with the specified K/V pair. If we're dealing >> with *lists* (of maps), then the 'lists' module is, at first glance, a >> reasonable place to put this functionality. > > The problem is that lists are such a general-purpose data structure. > Suppose we had an analogue of select(2) that took a list of io devices > and returned one of them that was ready, would we put that in the > lists: module or would we expect it to be in io:? > > For me, the important thing is not whether the elements are > held in a list or a tuple or a set or whatever but what are > the focal things we're acting on. > > Of course, this is arguably an instance of the expression problem, > and such there may be no best answer. > I agree that these functions should not be added to the lists module, since it already has many functions that do not need to be confused with newer additions (new functions confused with lists:keymap/3, lists:map/2, lists:mapfoldl/3, lists:mapfoldr/3, simply due to the word "map" that doesn't relate to map elements). I understand the lists module already has functions for handling lists that contain tuples with the "key" functions. In my mind, this argues for adding the concept of a namespace to Erlang source code, so that ".map:" or ".maps:" can be used as a suffix on lists, to show that the functions relate to lists that contain maps. I understand the past concept of Erlang packages failed and was gradually removed. Also, I understand there is the common response that you can always put periods in module names, similar to how elixir modules have an "Elixir." module name prefix for beam output. The current situation is unable to easily manage complexity as the SLOC of Erlang source code grows, artificially limiting the size of Erlang codebases, and the dependencies they may be able to manage (with the additional problem of being unable to utilize more than 1 version of the same dependency). I believe this hesitation to add more complexity to the single "lists" module is evidence of a problem, and the need for more hierarchical organization of source code, even if it only results in periods in module names. Best Regards, Michael From ok@REDACTED Tue Aug 22 05:01:03 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 22 Aug 2017 15:01:03 +1200 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> <820DD611-8E95-4372-B3DD-AF186FAC14F5@gmail.com> Message-ID: <66ae0714-f660-b0da-0885-019107cd2c81@cs.otago.ac.nz> On 22/08/17 4:18 AM, Lukas Larsson wrote: > I agree with Richard that lists:first/2[1] is missing. > [1]: though I think it should be called lists:find/2 for consistency > with lists:keyfind/3 and to not be confused as the opposite of lists:last/1 lists:first/2 is the opposite of lists:last/2. I thought I included that. If not, last(Pred, [X|Xs]) -> case Pred(X) of true -> last_aux(Pred, Xs, X) ; false -> last(Pred, Xs) end; last(_, []) -> error. last_aux(Pred, [X|Xs], Y) -> last_aux(Pred, Xs, case Pred(X) of true -> X ; false -> Y end); last_aux(_, [], Y) -> {ok, Y}. From max.lapshin@REDACTED Tue Aug 22 09:38:18 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 22 Aug 2017 10:38:18 +0300 Subject: [erlang-questions] lists:mapfind? In-Reply-To: <66ae0714-f660-b0da-0885-019107cd2c81@cs.otago.ac.nz> References: <774d5775-11bb-a4b9-e747-478990badaac@cs.otago.ac.nz> <820DD611-8E95-4372-B3DD-AF186FAC14F5@gmail.com> <66ae0714-f660-b0da-0885-019107cd2c81@cs.otago.ac.nz> Message-ID: Perhaps lists:keyfind can work with maps also? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Aug 22 09:57:46 2017 From: zxq9@REDACTED (zxq9) Date: Tue, 22 Aug 2017 16:57:46 +0900 Subject: [erlang-questions] lists:mapfind? In-Reply-To: References: <66ae0714-f660-b0da-0885-019107cd2c81@cs.otago.ac.nz> Message-ID: <2962729.HUqvovYIp1@changa> On 2017?08?22? ??? 10:38:18 Max Lapshin wrote: > Perhaps lists:keyfind can work with maps also? tl;dr: Please, no. Brace yourself: This really goes against the idea that maps are K/V thingies where K is a set and V is whatever. Lists of tuples is a VERY basic functional data structure -- to the point that, more or less, an entire language and paradigm was written based on it. A convenience function over that makes perfect sense if you need to figure out things about a list of maps. But this doesn't strike me as the underlying problem, really. This sounds a lot like someone is facing an annoying moment where their program's semantics have gotten the better of them and now they have got a list of maps that shouldn't be naked maps. They should be having a list of tuples something like [{MetaData, Map}] so that some useful meta about the map is conveyed outside of the map. Not EVERYTHING should be inside of a map. K/V types are really limited in what they are good for. Just about any data type can be shoehorned onto inappropriate use cases. I realize I'm getting ranty about this right now, but that's because I have spent a huge amount of time staring at semantically retarded code written by people who can only ever imagine that there is one data type in the whole language, maps, and they proceed to write reams of code for manipulating them, inspecting them, and so on when more boring, direct, semantically unambiguous code would have been dramatically better, faster, easier to debug, and more flexible (in the good ways -- "every function can take a map! hurray!" is NOT what "flexibility" is supposed to mean...). Hidden within the desire for such a function is a deep need for some data structure refactoring. I would feel this instinctively -- and so far that instict has never steered me wrong (though occasionally there is a borderline case where there is just no good tradeoff for some reason, at least not one that is obvious within the time allocated to study of the problem, or would imply a much deeper refactoring of the program because other stuff is all globbed together by intimate knowledge of a data structure or magical keys or something). -Craig From ola.a.andersson@REDACTED Tue Aug 22 10:42:01 2017 From: ola.a.andersson@REDACTED (Ola Andersson A) Date: Tue, 22 Aug 2017 08:42:01 +0000 Subject: [erlang-questions] lists:pmap ? Message-ID: Reading the discussion about list:mapfind reminded me of a function I rediscovered recently. It's rpc:pmap/3 that was originally defined for use in a distributed environment, spreading out the processing over several nodes. It actually works on a single multicore node as well even though it wasn't designed for that purpose. With the limited tests I have done it seems to significantly outperform lists:map/2 and also scale reasonably well. The almost negligible cost of spawning erlang processes is still amazing. How about adding a lists:pmap/2 function, designed for multicore, in the lists module? /OLA. From zxq9@REDACTED Tue Aug 22 10:58:47 2017 From: zxq9@REDACTED (zxq9) Date: Tue, 22 Aug 2017 17:58:47 +0900 Subject: [erlang-questions] lists:pmap ? In-Reply-To: References: Message-ID: <1657199.e4QDEPUJA7@changa> On 2017?08?22? ??? 08:42:01 Ola Andersson A wrote: > Reading the discussion about list:mapfind reminded me of a function I rediscovered recently. > It's rpc:pmap/3 that was originally defined for use in a distributed environment, spreading out the processing over several nodes. It actually works on a single multicore node as well even though it wasn't designed for that purpose. > With the limited tests I have done it seems to significantly outperform lists:map/2 and also scale reasonably well. The almost negligible cost of spawning erlang processes is still amazing. > How about adding a lists:pmap/2 function, designed for multicore, in the lists module? I've implemented something along these lines several times in an ad hoc manner for pure map functions in various projects, especially client-side code (but the function being mapped really needs to be pure!). I imagine that's fairly common (in client code not doing this is sometimes crazy). It WOULD be pretty awesome to have this in the standard library, and I can easily imagine a version of that which would be written against other collection-type data structures as well... But I also imagine that the arbitrariness of the input would suddenly make an implementation become non-trivial to go about in a safe way (or not at least without splashing the documentation liberally with warnings that pmap might blow your node up, that mapped functions really need to be pure, etc.). Do we limit worker spawns to a finite total size based on the VM's condition and keep up with it as return values are reported back? "The VM's condition" is a special ball of madness right there. Etc. This could be picked through, and probably quite effectively if some time is put to it. I just want to point out that the implementation will either be super naive (but still useful in cases where people know what they are dealing with), or super involved for what is conceptually a very simple idea -- but probably not very much happy middle exists between those two extremes. -Craig From z@REDACTED Tue Aug 22 12:50:14 2017 From: z@REDACTED (Danil Zagoskin) Date: Tue, 22 Aug 2017 13:50:14 +0300 Subject: [erlang-questions] Memory leak in SSL In-Reply-To: References: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> Message-ID: > Would it work for you if the test was changed to Size >= Max ? Yes. Yesterday I hot-loaded ssl_manager on affected system. Cache size slowly decreased to 1000 and stays in a range of 200-1100. On Mon, Aug 21, 2017 at 4:23 PM, Ingela Andin wrote: > Hi! > > 2017-08-21 13:36 GMT+02:00 Danil Zagoskin : > >> Hi Ingela! >> >> We've hit presumably the same bug on OTP 20. >> That's what I found: >> 1. This patch https://github.com/erlang/otp/ >> commit/256e01ce80b3aadd63f303b9bda5722ad313220f >> -- we start invalidation only on Size == Max >> 2. (flussonic@REDACTED)33> (sys:get_state(ssl_manager))#s >> tate.session_cache_server_max. >> > 1000 >> 3. (flussonic@REDACTED)37> redbug:start("ssl_session_cache:size/1 -> >> return", [{msgs, 2}]). >> ... >> ssl_session_cache:size/1 -> 8610379 >> >> Seems like on high load if we accept some new sessions while >> session_validation does its job, cache size limiting stops working. >> >> > Humm... maybe something was changed, that changed the criteria breaking > the assumption, let me ponder that some more. > Would it work for you if the test was changed to Size >= Max ? > > Regards Ingela Erlang/OTP Team - Ericsson AB > > > >> On Tue, Aug 8, 2017 at 4:55 PM, Ingela Andin >> wrote: >> >>> Hi! >>> >>> Depending on the version of the ssl application you may also set these >>> values. I guess you have an older version as they default to 1000. >>> >>> From doc: >>> >>> *"session_cache_client_max = integer() * >>> >>> Limits the growth of the clients session cache, that is how many >>> sessions towards servers that are cached to be used by new client >>> connections. If the maximum number of sessions is reached, the current >>> cache entries will be invalidated regardless of their remaining lifetime. >>> Defaults to 1000. >>> * session_cache_server_max = integer() * >>> >>> Limits the growth of the servers session cache, that is how many client >>> sessions are cached by the server. If the maximum number of sessions is >>> reached, the current cache entries will be invalidated regardless of their >>> remaining lifetime. Defaults to 1000." >>> >>> >>> Regards Ingela Erlang/OTP Team - Ericsson AB >>> >>> >>> >>> >>> 2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov : >>> >>>> Hello, >>>> >>>> Have you tried to disable reuse of ssl sessions? >>>> {ssl_options, [{reuse_sessions, false}]} >>>> >>>> And reduce the session time >>>> -ssl session_lifetime 120 >>>> >>>> Best Regards, >>>> Dmitry >>>> >>>> > On 8 Aug 2017, at 11.59, Max Treskin wrote: >>>> > >>>> > Hello, >>>> > >>>> > I have two different HTTPS-servers (different purposes, code, etc) >>>> built on top of cowboy/ranch, and both have memory leaks. >>>> > ETS table server_ssl_otp_session_cache has millions of records just >>>> after hours of work and consumes gigabytes of RAM. >>>> > What should I do to eliminate this? Is it bug or intended behaviour? >>>> > >>>> > Thanks >>>> > >>>> > >>>> > _______________________________________________ >>>> > erlang-questions mailing list >>>> > erlang-questions@REDACTED >>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> >> >> -- >> Danil Zagoskin | z@REDACTED >> > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Tue Aug 22 18:34:44 2017 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 22 Aug 2017 18:34:44 +0200 Subject: [erlang-questions] lists:pmap ? In-Reply-To: <1657199.e4QDEPUJA7@changa> References: <1657199.e4QDEPUJA7@changa> Message-ID: Interesting- I teach a class in concurrent and parallel programming where I show a simple pmap subject to the condition that all processes terminate within a reasonable time and with no errors. The more fun cases do things like limiting the number of parallel processes and versions which don't bother about the order of the replies in the result list. We might want to compute F(X) for all elements X in a list L, but given L = [X1,X2,X3] might not want to compute [F(X1),F(X2),F(X3)] any ordering might be acceptable (say [F(X2),F(X1),F(X3)]. If we have N cores we might want to limit the number of parallel processes to say 2N since we can't actually do more that N things in parallel - or we might not. You might want to look at this project http://skel.weebly.com/about-skel.html Which has a number of algorithmic skeletons for parallelizing erlang programs Cheers /Joe On Tue, Aug 22, 2017 at 10:58 AM, zxq9 wrote: > On 2017?08?22? ??? 08:42:01 Ola Andersson A wrote: >> Reading the discussion about list:mapfind reminded me of a function I rediscovered recently. >> It's rpc:pmap/3 that was originally defined for use in a distributed environment, spreading out the processing over several nodes. It actually works on a single multicore node as well even though it wasn't designed for that purpose. >> With the limited tests I have done it seems to significantly outperform lists:map/2 and also scale reasonably well. The almost negligible cost of spawning erlang processes is still amazing. >> How about adding a lists:pmap/2 function, designed for multicore, in the lists module? > > > I've implemented something along these lines several times in an ad hoc manner for pure map functions in various projects, especially client-side code (but the function being mapped really needs to be pure!). I imagine that's fairly common (in client code not doing this is sometimes crazy). > > It WOULD be pretty awesome to have this in the standard library, and I can easily imagine a version of that which would be written against other collection-type data structures as well... > > But I also imagine that the arbitrariness of the input would suddenly make an implementation become non-trivial to go about in a safe way (or not at least without splashing the documentation liberally with warnings that pmap might blow your node up, that mapped functions really need to be pure, etc.). Do we limit worker spawns to a finite total size based on the VM's condition and keep up with it as return values are reported back? "The VM's condition" is a special ball of madness right there. Etc. > > This could be picked through, and probably quite effectively if some time is put to it. I just want to point out that the implementation will either be super naive (but still useful in cases where people know what they are dealing with), or super involved for what is conceptually a very simple idea -- but probably not very much happy middle exists between those two extremes. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bchesneau@REDACTED Tue Aug 22 21:38:51 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 22 Aug 2017 21:38:51 +0200 Subject: [erlang-questions] find a nif resource by its nale Message-ID: <52A66511-F61D-4A37-8B4F-C0D8FF8E6B94@gmail.com> I like the idea of not passing a nif resource like ets does (ie passing a ref or a name). A idea on how this could be done in an efficient manner? I actually think to use an rw lock and a simple map in the private resource for it. But unsure what I should return on creation. Do we have to return the resource erlang term to keep it associated to the process? or simply creating it in the process will be enough? Beno?t From g@REDACTED Wed Aug 23 00:53:39 2017 From: g@REDACTED (Guilherme Andrade) Date: Tue, 22 Aug 2017 23:53:39 +0100 Subject: [erlang-questions] find a nif resource by its nale In-Reply-To: <52A66511-F61D-4A37-8B4F-C0D8FF8E6B94@gmail.com> References: <52A66511-F61D-4A37-8B4F-C0D8FF8E6B94@gmail.com> Message-ID: Perhaps you could store the resource in an ETS table, which would map names (arbitrary terms, atoms, ...) to the resource itself? It would require the extra dependency of having some process own the table, and one would have to be careful with leaks, but it should work. I think storing it in a C-land associative array could be done by always keeping at least one alive reference to it (e.g. by not calling 'enif_release_resource' while it the object lives), but avoiding resource leaks would be trickier at that point. There's a new 'enif_monitor_process' call available as of OTP 20, though; maybe one could simply make the objects owned by processes, monitor each of these processes and release the objects upon process death. I myself usually just wrap the 'enif_make_resource' result in some meaningful tagged tuple and return it so that it's debug-friendlier. On 22 August 2017 at 20:38, Benoit Chesneau wrote: > I like the idea of not passing a nif resource like ets does (ie passing a > ref or a name). A idea on how this could be done in an efficient manner? > > I actually think to use an rw lock and a simple map in the private > resource for it. But unsure what I should return on creation. Do we have to > return the resource erlang term to keep it associated to the process? or > simply creating it in the process will be enough? > > Beno?t > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Aug 23 07:59:12 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 23 Aug 2017 07:59:12 +0200 Subject: [erlang-questions] find a nif resource by its nale In-Reply-To: <52A66511-F61D-4A37-8B4F-C0D8FF8E6B94@gmail.com> References: <52A66511-F61D-4A37-8B4F-C0D8FF8E6B94@gmail.com> Message-ID: It is my understanding that in OTP 20, magic binaries are gone, and references are used instead: OTP-14205 Application(s): erts All uses of the magic binary kludge has been replaced by uses of erlang references. A magic binary was presented as an empty binary, but actually referred other data internally in the Erlang VM. Since they were presented as empty binaries, different magic binaries compared as equal, and also lost their internal data when passed out of an erlang node. The new usage of references has not got any of these strange semantic issues, and the usage of these references has been optimized to give the same performance benefits as well as memory usage benefits as magic binaries had. A couple of examples of previous uses of magic binaries are match specifications and NIF resources. So it sounds like you don't need to do anything? On 08/22/2017 09:38 PM, Benoit Chesneau wrote: > I like the idea of not passing a nif resource like ets does (ie passing a ref or a name). A idea on how this could be done in an efficient manner? > > I actually think to use an rw lock and a simple map in the private resource for it. But unsure what I should return on creation. Do we have to return the resource erlang term to keep it associated to the process? or simply creating it in the process will be enough? > > Beno?t > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From max.lapshin@REDACTED Wed Aug 23 11:53:15 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 23 Aug 2017 12:53:15 +0300 Subject: [erlang-questions] find a nif resource by its nale In-Reply-To: References: <52A66511-F61D-4A37-8B4F-C0D8FF8E6B94@gmail.com> Message-ID: Magic binary was very convenient to make some identification for the internal resource. Also it was the way to make mmap working (but mmap is a bad thing for reading from disk, so no problems) -------------- next part -------------- An HTML attachment was scrubbed... URL: From gattushivakrishna@REDACTED Wed Aug 23 12:27:33 2017 From: gattushivakrishna@REDACTED (Gattu shiva krishna) Date: Wed, 23 Aug 2017 15:57:33 +0530 Subject: [erlang-questions] Server Sent Events Message-ID: <8bacd4d7-7c25-4fe8-394a-c981ca638361@utl.in> Hi all, I'm learning to use yaws web-server and I came across server sent events which is used to send data to the client. To send data to the client I'm using the function yaws_sse:send_events(A#arg.clisock, Data). What is the format of arguments that need to be given to this function. I've googled enough and didn't get any satisfactory results. Below is a sample application where I'm trying to experiment on server-sent-events. Here in code I'm trying to create buttons dynamically in client side based on data that comes from server side. --------------------------------------------------------------------------------------------------------------------------------------- SERVER SIDE CODE: out(A) -> Data = yaws_sse:data(["Button1", "Button2","Button3","Button4"]), yaws_sse:send_events(A#arg.clisock, Data). --------------------------------------------------------------------------------------------------------------------------------------- CLIENT SIDE CODE: From jesper.louis.andersen@REDACTED Wed Aug 23 12:38:08 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 23 Aug 2017 10:38:08 +0000 Subject: [erlang-questions] Server Sent Events In-Reply-To: <8bacd4d7-7c25-4fe8-394a-c981ca638361@utl.in> References: <8bacd4d7-7c25-4fe8-394a-c981ca638361@utl.in> Message-ID: Here is what I think is going wrong, though I'm not entirely sure: The Yaws yaws_sse:data/1,2 functions assume the data you give them are so-called iodata(). So when you write something like ["Button1", "Button2"] and send it, it is effectively flattened into a binary <<"Button1Button2">> The iodata() system is really nice for doing scatter/gatter (typo intended) IO, but you have to invent some kind of framing in which you run. For example, we could use JSON: X = [<<"Button1">>, <<"Button"2>>], Data = yaws_sse:data(jsx:encode(X)), ... which encodes the data into a JSON structure which we can handle on the client side. Also, look up the iodata() concept. It is pretty nice. There is an old post by James Hague on it: http://prog21.dadgum.com/70.html On Wed, Aug 23, 2017 at 12:27 PM Gattu shiva krishna < gattushivakrishna@REDACTED> wrote: > Hi all, > > I'm learning to use yaws web-server and I came across server > sent events which is used to send data to the client. To send data to > the client I'm using the function yaws_sse:send_events(A#arg.clisock, > Data). What is the format of arguments that need to be given to this > function. I've googled enough and didn't get any satisfactory results. > Below is a sample application where I'm trying to experiment on > server-sent-events. > > Here in code I'm trying to create buttons dynamically in client > side based on data that comes from server side. > > > --------------------------------------------------------------------------------------------------------------------------------------- > > SERVER SIDE CODE: > > > out(A) -> > Data = yaws_sse:data(["Button1", > "Button2","Button3","Button4"]), > > yaws_sse:send_events(A#arg.clisock, Data). > > > > --------------------------------------------------------------------------------------------------------------------------------------- > > CLIENT SIDE CODE: > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Aug 23 13:07:00 2017 From: zxq9@REDACTED (zxq9) Date: Wed, 23 Aug 2017 20:07 +0900 Subject: [erlang-questions] lists:pmap ? In-Reply-To: References: <1657199.e4QDEPUJA7@changa> Message-ID: <2741893.AWqbQRXHSJ@changa> On 2017?08?22? ??? 18:34:44 you wrote: > You might want to look at this project > > http://skel.weebly.com/about-skel.html > > Which has a number of algorithmic skeletons for parallelizing erlang programs Hi, Joe! Thank you for pointing out Skel! I'll have more time to see how hard it would be to package it (or maybe something derived from it) soon. I imagine it might be an easy way to scratch the pmap-flavored itch Ola was talking about. -Craig From bevza@REDACTED Wed Aug 23 10:24:22 2017 From: bevza@REDACTED (Bevza, Leonid V.) Date: Wed, 23 Aug 2017 08:24:22 +0000 Subject: [erlang-questions] Erlang roadmap and TLS 1.3 support? Message-ID: Hi, everyone, I have the question about the roadmap of Erlang. Does erlang team have any plans to support TLS 1.3 in future? I am sure that you are going to support it in future, but maybe you have more precise plans? Thanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal@REDACTED Wed Aug 23 13:12:17 2017 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Wed, 23 Aug 2017 13:12:17 +0200 Subject: [erlang-questions] lists:pmap ? In-Reply-To: <2741893.AWqbQRXHSJ@changa> References: <1657199.e4QDEPUJA7@changa> <2741893.AWqbQRXHSJ@changa> Message-ID: Elixir recently introduced Task.async_stream/3,5 functions [1]. The primary feature is a pmap over a collection with a bounded concurrency. It's surprisingly useful and covers a lot of use cases in practice. An additional nice thing is that it accepts a potentially lazy collection and returns a lazy collection as well, so it's possible to consume only a part of it. Micha?. [1]:?https://hexdocs.pm/elixir/Task.html#async_stream/3 On 23 Aug 2017, 13:07 +0200, zxq9 , wrote: > On 2017?08?22? ??? 18:34:44 you wrote: > > You might want to look at this project > > > > http://skel.weebly.com/about-skel.html > > > > Which has a number of algorithmic skeletons for parallelizing erlang programs > > Hi, Joe! > > Thank you for pointing out Skel! > > I'll have more time to see how hard it would be to package it (or maybe something derived from it) soon. I imagine it might be an easy way to scratch the pmap-flavored itch Ola was talking about. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From gattushivakrishna@REDACTED Thu Aug 24 07:53:41 2017 From: gattushivakrishna@REDACTED (Gattu shiva krishna) Date: Thu, 24 Aug 2017 11:23:41 +0530 Subject: [erlang-questions] Server Sent Events In-Reply-To: References: <8bacd4d7-7c25-4fe8-394a-c981ca638361@utl.in> Message-ID: <30e6fae0-a25e-cbe3-3449-fa5a8c99042f@utl.in> Hi, when i'm using below function Data = yaws_sse:data( [ "button1", "button2" ] ), the result value of that function is [<<"data:">>, [ "button1", "button2" ], <<"\n">>] when i'm trying to send this data to client side using below function yaws_sse:send_events(A#arg.clisock, Data) but in client side iam unable to retrieve the data, Is there any solution for this. ---------------------------------------------------------------------------------------------------------------------------------------------- when i'm trying to use this function ( yaws_sse:send_events/2 )manually in yaws shell. for example : 1> yaws_sse:send_events( '#port<0.3333>', [<<"data:">>,[ "button1", "button2" ],<<"\n">>]). It is giving some error like this : ** exception error: no function clause matching gen_tcp:send('#port<0.3333>', [[<<"data:">>, [ "button1", "button2" ], <<"\n">>], <<"\n">>]) (gen_tcp.erl, line 261) What is the format of arguments that need to be given to this function. On Wednesday 23 August 2017 04:08 PM, Jesper Louis Andersen wrote: > Here is what I think is going wrong, though I'm not entirely sure: > > The Yaws yaws_sse:data/1,2 functions assume the data you give them are > so-called iodata(). So when you write something like > > ["Button1", "Button2"] > > and send it, it is effectively flattened into a binary > > <<"Button1Button2">> > > The iodata() system is really nice for doing scatter/gatter (typo > intended) IO, but you have to invent some kind of framing in which you > run. > > For example, we could use JSON: > > X = [<<"Button1">>, <<"Button"2>>], > Data = yaws_sse:data(jsx:encode(X)), > ... > > which encodes the data into a JSON structure which we can handle on > the client side. > > Also, look up the iodata() concept. It is pretty nice. There is an old > post by James Hague on it: > > http://prog21.dadgum.com/70.html > > > On Wed, Aug 23, 2017 at 12:27 PM Gattu shiva krishna > > wrote: > > Hi all, > > I'm learning to use yaws web-server and I came across server > sent events which is used to send data to the client. To send data to > the client I'm using the function yaws_sse:send_events(A#arg.clisock, > Data). What is the format of arguments that need to be given to this > function. I've googled enough and didn't get any satisfactory results. > Below is a sample application where I'm trying to experiment on > server-sent-events. > > Here in code I'm trying to create buttons dynamically in client > side based on data that comes from server side. > > --------------------------------------------------------------------------------------------------------------------------------------- > > SERVER SIDE CODE: > > > out(A) -> > Data = yaws_sse:data(["Button1", > "Button2","Button3","Button4"]), > > yaws_sse:send_events(A#arg.clisock, Data). > > > --------------------------------------------------------------------------------------------------------------------------------------- > > CLIENT SIDE CODE: > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Thu Aug 24 10:23:36 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 24 Aug 2017 10:23:36 +0200 Subject: [erlang-questions] Patch package OTP 20.0.3 released Message-ID: <20170824082336.GA85389@erix.ericsson.se> This is README can be found at: http://erlang.org/download/OTP-20.0.3.README As usual for source code patches there is no downloadable tarball nor windows installer. Patch Package: OTP 20.0.3 Git Tag: OTP-20.0.3 Date: 2017-08-24 Trouble Report Id: OTP-13011, OTP-14522, OTP-14524, OTP-14541 Seq num: ERIERL-48, ERIERL-60 System: OTP Release: 20 Application: asn1-5.0.2, compiler-7.1.1, erts-9.0.3, ssh-4.5.1 Predecessor: OTP 20.0.2 Check out the git tag OTP-20.0.3, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- asn1-5.0.2 ------------------------------------------------------ --------------------------------------------------------------------- The asn1-5.0.2 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-13011 Application(s): asn1 Related Id(s): ERIERL-60 Default values now work in extension for PER, so if you give the atom asn1_DEFAULT instead of a value it will become the default value. Full runtime dependencies of asn1-5.0.2: erts-7.0, kernel-3.0, stdlib-2.0 --------------------------------------------------------------------- --- compiler-7.1.1 -------------------------------------------------- --------------------------------------------------------------------- The compiler-7.1.1 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-14522 Application(s): compiler Related Id(s): ERIERL-48 Fail labels on guard BIFs weren't taken into account during an optimization pass, and a bug in the validation pass sometimes prevented this from being noticed when a fault occurred. Full runtime dependencies of compiler-7.1.1: crypto-3.6, erts-9.0, hipe-3.12, kernel-4.0, stdlib-2.5 --------------------------------------------------------------------- --- erts-9.0.3 ------------------------------------------------------ --------------------------------------------------------------------- The erts-9.0.3 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-14524 Application(s): erts Binary append operations did not check for overflow, resulting in nonsensical results when huge binaries were appended. Full runtime dependencies of erts-9.0.3: kernel-5.0, sasl-3.0.1, stdlib-3.0 --------------------------------------------------------------------- --- ssh-4.5.1 ------------------------------------------------------- --------------------------------------------------------------------- The ssh-4.5.1 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-14541 Application(s): ssh Related Id(s): EIRERL-63 All unknown options are sent to the transport handler regardless of type. Full runtime dependencies of ssh-4.5.1: crypto-3.7.3, erts-6.0, kernel-3.0, public_key-1.4, stdlib-3.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From ingela.andin@REDACTED Thu Aug 24 11:16:57 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 24 Aug 2017 11:16:57 +0200 Subject: [erlang-questions] Memory leak in SSL In-Reply-To: References: <60BC7455-8A6D-457A-993E-6FE70CECFD59@gmail.com> Message-ID: Hi! 2017-08-22 12:50 GMT+02:00 Danil Zagoskin : > > Would it work for you if the test was changed to Size >= Max ? > > Yes. Yesterday I hot-loaded ssl_manager on affected system. Cache size > slowly decreased to 1000 and stays in a range of 200-1100. > > And it now makes sense to me, I have included the fix in our builds. It probably will fix a wobbling test case. Scheduled for inclusion in OTP 20.1 Regards Ingela Erlang/OTP team - Ericsson AB > On Mon, Aug 21, 2017 at 4:23 PM, Ingela Andin > wrote: > >> Hi! >> >> 2017-08-21 13:36 GMT+02:00 Danil Zagoskin : >> >>> Hi Ingela! >>> >>> We've hit presumably the same bug on OTP 20. >>> That's what I found: >>> 1. This patch https://github.com/erlang/otp/ >>> commit/256e01ce80b3aadd63f303b9bda5722ad313220f >>> -- we start invalidation only on Size == Max >>> 2. (flussonic@REDACTED)33> (sys:get_state(ssl_manager))#s >>> tate.session_cache_server_max. >>> > 1000 >>> 3. (flussonic@REDACTED)37> redbug:start("ssl_session_cache:size/1 -> >>> return", [{msgs, 2}]). >>> ... >>> ssl_session_cache:size/1 -> 8610379 >>> >>> Seems like on high load if we accept some new sessions while >>> session_validation does its job, cache size limiting stops working. >>> >>> >> Humm... maybe something was changed, that changed the criteria breaking >> the assumption, let me ponder that some more. >> Would it work for you if the test was changed to Size >= Max ? >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> >> >>> On Tue, Aug 8, 2017 at 4:55 PM, Ingela Andin >>> wrote: >>> >>>> Hi! >>>> >>>> Depending on the version of the ssl application you may also set these >>>> values. I guess you have an older version as they default to 1000. >>>> >>>> From doc: >>>> >>>> *"session_cache_client_max = integer() * >>>> >>>> Limits the growth of the clients session cache, that is how many >>>> sessions towards servers that are cached to be used by new client >>>> connections. If the maximum number of sessions is reached, the current >>>> cache entries will be invalidated regardless of their remaining lifetime. >>>> Defaults to 1000. >>>> * session_cache_server_max = integer() * >>>> >>>> Limits the growth of the servers session cache, that is how many client >>>> sessions are cached by the server. If the maximum number of sessions is >>>> reached, the current cache entries will be invalidated regardless of their >>>> remaining lifetime. Defaults to 1000." >>>> >>>> >>>> Regards Ingela Erlang/OTP Team - Ericsson AB >>>> >>>> >>>> >>>> >>>> 2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov : >>>> >>>>> Hello, >>>>> >>>>> Have you tried to disable reuse of ssl sessions? >>>>> {ssl_options, [{reuse_sessions, false}]} >>>>> >>>>> And reduce the session time >>>>> -ssl session_lifetime 120 >>>>> >>>>> Best Regards, >>>>> Dmitry >>>>> >>>>> > On 8 Aug 2017, at 11.59, Max Treskin wrote: >>>>> > >>>>> > Hello, >>>>> > >>>>> > I have two different HTTPS-servers (different purposes, code, etc) >>>>> built on top of cowboy/ranch, and both have memory leaks. >>>>> > ETS table server_ssl_otp_session_cache has millions of records just >>>>> after hours of work and consumes gigabytes of RAM. >>>>> > What should I do to eliminate this? Is it bug or intended behaviour? >>>>> > >>>>> > Thanks >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > erlang-questions mailing list >>>>> > erlang-questions@REDACTED >>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >>> >>> -- >>> Danil Zagoskin | z@REDACTED >>> >> >> > > > -- > Danil Zagoskin | z@REDACTED > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Thu Aug 24 15:10:19 2017 From: vinoski@REDACTED (Steve Vinoski) Date: Thu, 24 Aug 2017 09:10:19 -0400 Subject: [erlang-questions] Server Sent Events In-Reply-To: <30e6fae0-a25e-cbe3-3449-fa5a8c99042f@utl.in> References: <8bacd4d7-7c25-4fe8-394a-c981ca638361@utl.in> <30e6fae0-a25e-cbe3-3449-fa5a8c99042f@utl.in> Message-ID: On Thu, Aug 24, 2017 at 1:53 AM, Gattu shiva krishna < gattushivakrishna@REDACTED> wrote: > Hi, > > when i'm using below function > > Data = yaws_sse:data( [ "button1", "button2" ] ), > > the result value of that function is > > [<<"data:">>, [ "button1", "button2" ], <<"\n">>] > As Jesper already explained, what the client will receive in this case is the equivalent of <<"data:button1button2\n">> Is that really what you're trying to send? Parsing that seems difficult. > when i'm trying to send this data to client side using below > function > yaws_sse:send_events(A#arg.clisock, Data) > > but in client side iam unable to retrieve the data, Is there any > solution for this. > > ------------------------------------------------------------ > ------------------------------------------------------------ > ---------------------- > > when i'm trying to use this function ( yaws_sse:send_events/2 > )manually in yaws shell. > > for example : > > 1> yaws_sse:send_events( '#port<0.3333>', [<<"data:">>,[ > "button1", "button2" ],<<"\n">>]). > > It is giving some error like this : > > ** exception error: no function clause matching > gen_tcp:send('#port<0.3333>', > [[<<"data:">>, > [ "button1", "button2" ], > <<"\n">>], > <<"\n">>]) (gen_tcp.erl, line 261) > > What is the format of arguments that need to be given to this function. > You're getting this error because you're passing the port argument as an atom, not as a port. Assuming you're using a relatively recent version of Erlang/OTP, in the shell the port you've mentioned should be specified as #Port<0.3333> without quotes. The whole call becomes yaws_sse:send_events(#Port<0.3333>, [<<"data:">>,[ "button1", "button2" ],<<"\n">>]). or more simply but equivalently yaws_sse:send_events(#Port<0.3333>, <<"data:button1button2\n">>). which will send the data you specify, assuming #Port<0.3333> is really the right connection. But again, the data you're sending is poorly structured and so won't be easy for your client to deal with. --steve > On Wednesday 23 August 2017 04:08 PM, Jesper Louis Andersen wrote: > > Here is what I think is going wrong, though I'm not entirely sure: > > The Yaws yaws_sse:data/1,2 functions assume the data you give them are > so-called iodata(). So when you write something like > > ["Button1", "Button2"] > > and send it, it is effectively flattened into a binary > > <<"Button1Button2">> > > The iodata() system is really nice for doing scatter/gatter (typo > intended) IO, but you have to invent some kind of framing in which you run. > > For example, we could use JSON: > > X = [<<"Button1">>, <<"Button"2>>], > Data = yaws_sse:data(jsx:encode(X)), > ... > > which encodes the data into a JSON structure which we can handle on the > client side. > > Also, look up the iodata() concept. It is pretty nice. There is an old > post by James Hague on it: > > http://prog21.dadgum.com/70.html > > > On Wed, Aug 23, 2017 at 12:27 PM Gattu shiva krishna < > gattushivakrishna@REDACTED> wrote: > >> Hi all, >> >> I'm learning to use yaws web-server and I came across server >> sent events which is used to send data to the client. To send data to >> the client I'm using the function yaws_sse:send_events(A#arg.clisock, >> Data). What is the format of arguments that need to be given to this >> function. I've googled enough and didn't get any satisfactory results. >> Below is a sample application where I'm trying to experiment on >> server-sent-events. >> >> Here in code I'm trying to create buttons dynamically in client >> side based on data that comes from server side. >> >> ------------------------------------------------------------ >> ------------------------------------------------------------ >> --------------- >> >> SERVER SIDE CODE: >> >> >> out(A) -> >> Data = yaws_sse:data(["Button1", >> "Button2","Button3","Button4"]), >> >> yaws_sse:send_events(A#arg.clisock, Data). >> >> >> ------------------------------------------------------------ >> ------------------------------------------------------------ >> --------------- >> >> CLIENT SIDE CODE: >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Aug 24 21:59:20 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 24 Aug 2017 21:59:20 +0200 Subject: [erlang-questions] Erlang roadmap and TLS 1.3 support? In-Reply-To: References: Message-ID: Hi! We will definitely take steps towards supporting TLS 1.3 during the development of OTP 21. But there are too many unknown factors that say anything certain. Regards Ingela Erlang/OTP team - Ericsson AB 2017-08-23 10:24 GMT+02:00 Bevza, Leonid V. : > Hi, everyone, > > > > I have the question about the roadmap of Erlang. Does erlang team have any > plans to support TLS 1.3 in future? I am sure that you are going to support > it in future, but maybe you have more precise plans? > > > > Thanks in advance. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Aug 24 22:12:39 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 24 Aug 2017 22:12:39 +0200 Subject: [erlang-questions] find a nif resource by its nale In-Reply-To: References: <52A66511-F61D-4A37-8B4F-C0D8FF8E6B94@gmail.com> Message-ID: Thanks for the answer :) I also map it in a tuple or record. I think the main idea of not having it in ETS is to remove the need of setup a another ETS table and remove some roundtrips in the process. I forgot about the liveness of the resource anyway, good call... I wish i could could use latest erlang 20 addition for it but it's not possible yet ... So I guess I will stick to ETS for now. On Wed, Aug 23, 2017 at 12:53 AM, Guilherme Andrade wrote: > Perhaps you could store the resource in an ETS table, which would map > names (arbitrary terms, atoms, ...) to the resource itself? It would > require the extra dependency of having some process own the table, and one > would have to be careful with leaks, but it should work. > > I think storing it in a C-land associative array could be done by always > keeping at least one alive reference to it (e.g. by not calling > 'enif_release_resource' while it the object lives), but avoiding resource > leaks would be trickier at that point. There's a new 'enif_monitor_process' > call available as of OTP 20, though; maybe one could simply make the > objects owned by processes, monitor each of these processes and release the > objects upon process death. > > I myself usually just wrap the 'enif_make_resource' result in some > meaningful tagged tuple and return it so that it's debug-friendlier. > > On 22 August 2017 at 20:38, Benoit Chesneau wrote: > >> I like the idea of not passing a nif resource like ets does (ie passing a >> ref or a name). A idea on how this could be done in an efficient manner? >> >> I actually think to use an rw lock and a simple map in the private >> resource for it. But unsure what I should return on creation. Do we have to >> return the resource erlang term to keep it associated to the process? or >> simply creating it in the process will be enough? >> >> Beno?t >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Guilherme > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Aug 24 22:13:04 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 24 Aug 2017 22:13:04 +0200 Subject: [erlang-questions] find a nif resource by its nale In-Reply-To: References: <52A66511-F61D-4A37-8B4F-C0D8FF8E6B94@gmail.com> Message-ID: mm not sure I follow. did you mixed the threads? On Wed, Aug 23, 2017 at 11:53 AM, Max Lapshin wrote: > Magic binary was very convenient to make some identification for the > internal resource. > > Also it was the way to make mmap working (but mmap is a bad thing for > reading from disk, so no problems) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From liuzhongzheng2012@REDACTED Fri Aug 25 04:06:40 2017 From: liuzhongzheng2012@REDACTED (Zhongzheng Liu) Date: Fri, 25 Aug 2017 10:06:40 +0800 Subject: [erlang-questions] Non-blocking for gen_tcp:accept? Message-ID: Hi all: Socket can be set to {active, true} to receive data by message instead of using gen_tcp:recv. But there is no {active_accept, true} option to make accept non-blocking. We have to use the blocking function gen_tcp:accept, why? Is it a good idea to deliver a message like {tcp, accept, Socket} to the accepting process when a connection is coming? Thanks Liu Zhongzheng From duncan@REDACTED Fri Aug 25 04:36:04 2017 From: duncan@REDACTED (duncan@REDACTED) Date: Thu, 24 Aug 2017 19:36:04 -0700 Subject: [erlang-questions] OASIS SARIF - standardizing format of static analysis tools (eg dialyzer) Message-ID: <20170824193604.7e43b23f706d1a78218bd3e1c66e57ee.3d63f3ac68.wbe@email23.godaddy.com> OASIS is forming a new group to standardize the output of static analysis tools. I presume this may impact (and be an opportunity for) tools like dialyzer. Is anyone planning to attend? I may attend but I'm not sure I'm expert enough to represent erlang's interests. Info can be found at: - https://www.oasis-open.org/committees/sarif/charter.php - https://lists.oasis-open.org/archives/sarif/201708/msg00000.html (first meeting 6-Sep) Duncan Sparrell sFractal Consulting LLC iPhone, iTypo, iApologize From vances@REDACTED Fri Aug 25 08:48:21 2017 From: vances@REDACTED (Vance Shipley) Date: Fri, 25 Aug 2017 12:18:21 +0530 Subject: [erlang-questions] Non-blocking for gen_tcp:accept? In-Reply-To: References: Message-ID: On Fri, Aug 25, 2017 at 7:36 AM, Zhongzheng Liu wrote: > Socket can be set to {active, true} to receive data by message instead > of using gen_tcp:recv. > > But there is no {active_accept, true} option to make accept > non-blocking. We have to use the blocking function gen_tcp:accept, > why? > > Is it a good idea to deliver a message like {tcp, accept, Socket} to > the accepting process when a connection is coming? I certainly think so. I use an undocumented method described here: http://www2.erlangcentral.org/wiki/?title=Building_a_Non-blocking_TCP_server_using_OTP_principles#Listener_Process Of course as it's undocumented you never know when a new version of OTP will break it which did in fact happen in a recent release. The fix is trivial if you compare the new version to the old. How about making non-blocking supported folks? -- -Vance From solvip@REDACTED Fri Aug 25 11:09:41 2017 From: solvip@REDACTED (=?UTF-8?B?U8O2bHZpIFDDoWxsIMOBc2dlaXJzc29u?=) Date: Fri, 25 Aug 2017 09:09:41 +0000 Subject: [erlang-questions] Non-blocking for gen_tcp:accept? In-Reply-To: References: Message-ID: A simple workaround is to accept in an out of band process and send the socket over to the process wanting non-blocking semantics. Something like the following: case gen_tcp:listen(...) of {ok, Listener} -> Parent = self(), Acceptor = fun() -> {ok, ClientSock} = gen_tcp:accept(Listener),, ok = gen_tcp:controlling_process(ClientSock, Parent), Parent ! {self(), tcp_accepted, ClientSock} end, Pid = spawn_link(Acceptor), ... Cheers S?lvi On Fri, Aug 25, 2017 at 6:48 AM, Vance Shipley wrote: > On Fri, Aug 25, 2017 at 7:36 AM, Zhongzheng Liu > wrote: >> Socket can be set to {active, true} to receive data by message instead >> of using gen_tcp:recv. >> >> But there is no {active_accept, true} option to make accept >> non-blocking. We have to use the blocking function gen_tcp:accept, >> why? >> >> Is it a good idea to deliver a message like {tcp, accept, Socket} to >> the accepting process when a connection is coming? > > I certainly think so. I use an undocumented method described here: > http://www2.erlangcentral.org/wiki/?title=Building_a_Non-blocking_TCP_server_using_OTP_principles#Listener_Process > > Of course as it's undocumented you never know when a new version of > OTP will break it which did in fact happen in a recent release. The > fix is trivial if you compare the new version to the old. > > How about making non-blocking supported folks? > > -- > -Vance > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From henrik.x.nord@REDACTED Fri Aug 25 14:20:31 2017 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Fri, 25 Aug 2017 12:20:31 +0000 Subject: [erlang-questions] Patch package OTP 20.0.4 released Message-ID: <1503663630.6990.0.camel@ericsson.com> Patch Package:???????????OTP 20.0.4 Git Tag:?????????????????OTP-20.0.4 Date:????????????????????2017-08-25 Trouble Report Id:???????OTP-14548, OTP-14554, OTP-14558 Seq num:?????????????????ERIERL-53, ERL-468 System:??????????????????OTP Release:?????????????????20 Application:?????????????dialyzer-3.2.1, erts-9.0.4 Predecessor:?????????????OTP 20.0.3 ?Check out the git tag OTP-20.0.4, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- dialyzer-3.2.1 -------------------------------------------------- ?--------------------------------------------------------------------- ?The dialyzer-3.2.1 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-14558????Application(s): dialyzer ???????????????Related Id(s): ERIERL-53 ???????????????Fix a bug where merging PLT:s could lose info. The bug ???????????????was introduced in Erlang/OTP 20.0. ?Full runtime dependencies of dialyzer-3.2.1: compiler-7.0, erts-8.0, ?hipe-3.15.4, kernel-5.0, stdlib-3.0, syntax_tools-2.0, wx-1.2 ?--------------------------------------------------------------------- ?--- erts-9.0.4 ------------------------------------------------------ ?--------------------------------------------------------------------- ?The erts-9.0.4 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-14548????Application(s): erts ???????????????Related Id(s): ERL-468, OTP-11997 ???????????????A timer internal bit-field used for storing scheduler ???????????????id was too small. As a result, VM internal timer data ???????????????structures could become inconsistent when using 1024 ???????????????schedulers on the system. Note that systems with less ???????????????than 1024 schedulers are not effected by this bug. ???????????????This bug was introduced in ERTS version 7.0 (OTP 18.0). ? OTP-14554????Application(s): erts ???????????????Related Id(s): ERL-468, OTP-14356 ???????????????Automatic cleanup of a BIF timer, when the owner ???????????????process terminated, could race with the timeout of the ???????????????timer. This could cause the VM internal data structures ???????????????to become inconsistent which very likely caused a VM ???????????????crash. ???????????????This bug was introduced in ERTS version 9.0 (OTP 20.0). ?Full runtime dependencies of erts-9.0.4: kernel-5.0, sasl-3.0.1, ?stdlib-3.0 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- From vladdu55@REDACTED Sun Aug 27 17:09:18 2017 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Sun, 27 Aug 2017 17:09:18 +0200 Subject: [erlang-questions] Publishing releases on erlang.org Message-ID: Hi! The latest package published on http://erlang.org/download/ is 20.0, despite having the 20.0.x README files there. This makes it difficult to use kerl, that gets the sources from there, and the original 20.0.0 has at least one big bug that was fixed in the meantime (string:len/1). Could you please publish the latest archive? Thanks! best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Aug 27 17:17:25 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Sun, 27 Aug 2017 17:17:25 +0200 Subject: [erlang-questions] Publishing releases on erlang.org In-Reply-To: References: Message-ID: kerl build git https://github.com/erlang/otp OTP-20.0.4 OTP-20.0.4 Just saying. :-) On 08/27/2017 05:09 PM, Vlad Dumitrescu wrote: > Hi! > > The latest package published on http://erlang.org/download/ is 20.0, > despite having the 20.0.x README files there. This makes it difficult to > use kerl, that gets the sources from there, and the original 20.0.0 has > at least one big bug that was fixed in the meantime (string:len/1). > > Could you please publish the latest archive? Thanks! > > best regards, > Vlad > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From vladdu55@REDACTED Sun Aug 27 17:18:50 2017 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Sun, 27 Aug 2017 17:18:50 +0200 Subject: [erlang-questions] Publishing releases on erlang.org In-Reply-To: References: Message-ID: Ah, okay, I never used the "git" option, didn't even know about it. Thanks a lot! regards, Vlad On Sun, Aug 27, 2017 at 5:17 PM, Lo?c Hoguin wrote: > kerl build git https://github.com/erlang/otp OTP-20.0.4 OTP-20.0.4 > > Just saying. :-) > > > On 08/27/2017 05:09 PM, Vlad Dumitrescu wrote: > >> Hi! >> >> The latest package published on http://erlang.org/download/ is 20.0, >> despite having the 20.0.x README files there. This makes it difficult to >> use kerl, that gets the sources from there, and the original 20.0.0 has at >> least one big bug that was fixed in the meantime (string:len/1). >> >> Could you please publish the latest archive? Thanks! >> >> best regards, >> Vlad >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > https://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Mon Aug 28 12:06:41 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 28 Aug 2017 12:06:41 +0200 Subject: [erlang-questions] Publishing releases on erlang.org In-Reply-To: References: Message-ID: <20170828100641.GA78051@erix.ericsson.se> On Sun, Aug 27, 2017 at 05:09:18PM +0200, Vlad Dumitrescu wrote: > Hi! > > The latest package published on http://erlang.org/download/ is 20.0, > despite having the 20.0.x README files there. This makes it difficult to > use kerl, that gets the sources from there, and the original 20.0.0 has at > least one big bug that was fixed in the meantime (string:len/1). > > Could you please publish the latest archive? Thanks! We do not publish tar archives for patch versions, so you will have to get the sources through the version control system (Git). > > best regards, > Vlad -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From kenji@REDACTED Tue Aug 29 03:53:17 2017 From: kenji@REDACTED (Kenji Rikitake) Date: Tue, 29 Aug 2017 10:53:17 +0900 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 Message-ID: I finally reviewed OTP rand module change between 19.3 and 20.0. While most of users won't recognize the changes, the details are considerably different. A good news is that if you're not aware of the PRNG algorithm you can keep the code untouched. The new default algorithm works ok and even better than the OTP 19 with less bugs. On the other hand, if you explicitly specify the algorithm in rand:seed/1, you are suggested to use a newly-available algorithms described in the rand module manual of OTP 20.0. Actually all the old (OTP 18 and 19) algorithms are now deprecated. See rand module manual of OTP 20. Use exrop (default of OTP 20) in most cases; if you use exs1024 that should be changed to exs1024s; if you stick with exsplus (default until OTP 19), use exsp instead. If you use other algorithms, consider converting to exrop. Also, if your code depends on the output range of rand:uniform/1 or rand:uniform_s/1, it has been changed as follows: OTP 18 and 19: 0.0 < X < 1.0 OTP 20: 0.0 =< X < 1.0 (note the =< operator) where X is the output of the functions. In short, since OTP 20, the functions may output 0.0. I noticed the changes shortly after OTP 20.0 release. These changes were in the last minute (19-MAR-2017 - 26-APR-2017) just before the OTP 20 release and after my Erlang and Elixir Factory 2017 trip so I didn't notice. I was not notified at all by Raimo Niskanen and other contributors about these changed either, so I had to take time and chance to review the code again. I know OTP is open source software and the OTP Team can modify the code for better features and removing bugs without notifying to anyone, so nobody is to blame on these changes. I'm assured that Raimo and the other contributors have done a great job on changing the details while maintaining the compatibility and fixing the trivial bugs which I left in the code. I would also like to note that crypto module also utilizes the rand module API to simplify access to cryptographically strong random number module since OTP 20. See crypto:rand_seed/{0,1} and crypto:rand_uniform/2. This is a good example to make use of rand module's extensible plugins. Thanks to Tuncer Ayaz for giving me a chance to review this. And always remember: if you are still dependent on random module, migrate to rand module now. Regards, Kenji Rikitake -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Tue Aug 29 10:35:28 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 29 Aug 2017 10:35:28 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: Message-ID: <20170829083528.GA91917@erix.ericsson.se> Thank you Kenji for reviewing the changes and summarizing the implications. Sorry about not getting you into the loop during the rewrite! As you guessed I was focused on plugging in 'crypto', fixing the flaws, incorporating the new algorithm from Prof. Vignia and keeping it as backwards compatible as possible, so I forgot about you... Regarding the changed uniform float behaviour: it is the functions rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now (OTP-20.0) documented to return 0.0 =< X < 1.0. Previously they _could_ return exactly 0.0, even though not stated in the documentation. But the probability for exactly 0.0 was about 2048 times less than it is now. This is because the float generation has been fixed to generate equidistant floats on the form K * 2^-53 so exactly 0.0 has now got the same probability as all other possible float values. I'd rather say that the documentation of rand:uniform/0 and rand:uniform_s/1 has been corrected to match their behaviour, and that the probability for exactly 0.0 has increased from about 1/64 to about 1/53. Despite this the distribution of generated numbers has actually not changed - it is still uniform over the range 0.0 =< X < 1.0. That is my view of the float changes, of which I am fairly certain. :-) / Raimo On Tue, Aug 29, 2017 at 10:53:17AM +0900, Kenji Rikitake wrote: > I finally reviewed OTP rand module change between 19.3 and 20.0. > While most of users won't recognize the changes, > the details are considerably different. > > A good news is that if you're not aware of the PRNG algorithm > you can keep the code untouched. The new default algorithm > works ok and even better than the OTP 19 with less bugs. > > On the other hand, if you explicitly specify the algorithm in rand:seed/1, > you are suggested to use a newly-available algorithms > described in the rand module manual of OTP 20.0. Actually > all the old (OTP 18 and 19) algorithms are now deprecated. > See rand module manual of OTP 20. Use exrop (default of OTP 20) in most > cases; > if you use exs1024 that should be changed to exs1024s; > if you stick with exsplus (default until OTP 19), use exsp instead. > If you use other algorithms, consider converting to exrop. > > Also, if your code depends on the output range of > rand:uniform/1 or rand:uniform_s/1, it has been changed as follows: > OTP 18 and 19: 0.0 < X < 1.0 > OTP 20: 0.0 =< X < 1.0 (note the =< operator) > where X is the output of the functions. > In short, since OTP 20, the functions may output 0.0. > > I noticed the changes shortly after OTP 20.0 release. > These changes were in the last minute (19-MAR-2017 - 26-APR-2017) > just before the OTP 20 release > and after my Erlang and Elixir Factory 2017 trip so I didn't notice. > I was not notified at all by Raimo Niskanen and other contributors > about these changed either, > so I had to take time and chance to review the code again. > I know OTP is open source software and the OTP Team can modify > the code for better features and removing bugs without notifying to anyone, > so nobody is to blame on these changes. > I'm assured that Raimo and the other contributors have done a great job > on changing the details while maintaining the compatibility > and fixing the trivial bugs which I left in the code. > > I would also like to note that crypto module also utilizes > the rand module API to simplify access to cryptographically strong > random number module since OTP 20. See crypto:rand_seed/{0,1} > and crypto:rand_uniform/2. This is a good example to make use of > rand module's extensible plugins. > > Thanks to Tuncer Ayaz for giving me a chance to review this. > > And always remember: if you are still dependent on random module, > migrate to rand module now. > > Regards, > Kenji Rikitake > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From raimo+erlang-questions@REDACTED Tue Aug 29 10:45:51 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 29 Aug 2017 10:45:51 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170829083528.GA91917@erix.ericsson.se> References: <20170829083528.GA91917@erix.ericsson.se> Message-ID: <20170829084551.GA2878@erix.ericsson.se> On Tue, Aug 29, 2017 at 10:35:28AM +0200, Raimo Niskanen wrote: > Thank you Kenji for reviewing the changes and summarizing the implications. > > Sorry about not getting you into the loop during the rewrite! > > As you guessed I was focused on plugging in 'crypto', fixing the flaws, > incorporating the new algorithm from Prof. Vignia and keeping it > as backwards compatible as possible, so I forgot about you... > > > > Regarding the changed uniform float behaviour: it is the functions > rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously > (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now > (OTP-20.0) documented to return 0.0 =< X < 1.0. > > Previously they _could_ return exactly 0.0, even though not stated in the > documentation. But the probability for exactly 0.0 was about 2048 times > less than it is now. This is because the float generation has been fixed > to generate equidistant floats on the form K * 2^-53 so exactly 0.0 has now > got the same probability as all other possible float values. > > I'd rather say that the documentation of rand:uniform/0 and rand:uniform_s/1 > has been corrected to match their behaviour, and that the probability for > exactly 0.0 has increased from about 1/64 to about 1/53. Sorry! That should be: from about 1/2^64 to about 1/2^53. / Raimo > > Despite this the distribution of generated numbers has actually not changed > - it is still uniform over the range 0.0 =< X < 1.0. > > That is my view of the float changes, of which I am fairly certain. :-) > > > / Raimo > > > > On Tue, Aug 29, 2017 at 10:53:17AM +0900, Kenji Rikitake wrote: > > I finally reviewed OTP rand module change between 19.3 and 20.0. > > While most of users won't recognize the changes, > > the details are considerably different. > > > > A good news is that if you're not aware of the PRNG algorithm > > you can keep the code untouched. The new default algorithm > > works ok and even better than the OTP 19 with less bugs. > > > > On the other hand, if you explicitly specify the algorithm in rand:seed/1, > > you are suggested to use a newly-available algorithms > > described in the rand module manual of OTP 20.0. Actually > > all the old (OTP 18 and 19) algorithms are now deprecated. > > See rand module manual of OTP 20. Use exrop (default of OTP 20) in most > > cases; > > if you use exs1024 that should be changed to exs1024s; > > if you stick with exsplus (default until OTP 19), use exsp instead. > > If you use other algorithms, consider converting to exrop. > > > > Also, if your code depends on the output range of > > rand:uniform/1 or rand:uniform_s/1, it has been changed as follows: > > OTP 18 and 19: 0.0 < X < 1.0 > > OTP 20: 0.0 =< X < 1.0 (note the =< operator) > > where X is the output of the functions. > > In short, since OTP 20, the functions may output 0.0. > > > > I noticed the changes shortly after OTP 20.0 release. > > These changes were in the last minute (19-MAR-2017 - 26-APR-2017) > > just before the OTP 20 release > > and after my Erlang and Elixir Factory 2017 trip so I didn't notice. > > I was not notified at all by Raimo Niskanen and other contributors > > about these changed either, > > so I had to take time and chance to review the code again. > > I know OTP is open source software and the OTP Team can modify > > the code for better features and removing bugs without notifying to anyone, > > so nobody is to blame on these changes. > > I'm assured that Raimo and the other contributors have done a great job > > on changing the details while maintaining the compatibility > > and fixing the trivial bugs which I left in the code. > > > > I would also like to note that crypto module also utilizes > > the rand module API to simplify access to cryptographically strong > > random number module since OTP 20. See crypto:rand_seed/{0,1} > > and crypto:rand_uniform/2. This is a good example to make use of > > rand module's extensible plugins. > > > > Thanks to Tuncer Ayaz for giving me a chance to review this. > > > > And always remember: if you are still dependent on random module, > > migrate to rand module now. > > > > Regards, > > Kenji Rikitake > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From raimo+erlang-questions@REDACTED Tue Aug 29 13:22:56 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 29 Aug 2017 13:22:56 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170829084551.GA2878@erix.ericsson.se> References: <20170829083528.GA91917@erix.ericsson.se> <20170829084551.GA2878@erix.ericsson.se> Message-ID: <20170829112256.GA3559@erix.ericsson.se> On Tue, Aug 29, 2017 at 10:45:51AM +0200, Raimo Niskanen wrote: > On Tue, Aug 29, 2017 at 10:35:28AM +0200, Raimo Niskanen wrote: > > Thank you Kenji for reviewing the changes and summarizing the implications. > > > > Sorry about not getting you into the loop during the rewrite! > > > > As you guessed I was focused on plugging in 'crypto', fixing the flaws, > > incorporating the new algorithm from Prof. Vignia and keeping it > > as backwards compatible as possible, so I forgot about you... > > > > > > > > Regarding the changed uniform float behaviour: it is the functions > > rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously > > (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now > > (OTP-20.0) documented to return 0.0 =< X < 1.0. > > > > Previously they _could_ return exactly 0.0, even though not stated in the > > documentation. But the probability for exactly 0.0 was about 2048 times > > less than it is now. This is because the float generation has been fixed > > to generate equidistant floats on the form K * 2^-53 so exactly 0.0 has now > > got the same probability as all other possible float values. > > > > I'd rather say that the documentation of rand:uniform/0 and rand:uniform_s/1 > > has been corrected to match their behaviour, and that the probability for > > exactly 0.0 has increased from about 1/64 to about 1/53. > > Sorry! That should be: from about 1/2^64 to about 1/2^53. And that was for the 64-bit generators. For the _default_ 58-bit generator the probability for exactly 0.0 has changed from about 1/2^58 to about 1/2^53 i.e increased with a factor 32. / Raimo > > / Raimo > > > > > > Despite this the distribution of generated numbers has actually not changed > > - it is still uniform over the range 0.0 =< X < 1.0. > > > > That is my view of the float changes, of which I am fairly certain. :-) > > > > > > / Raimo > > > > > > > > On Tue, Aug 29, 2017 at 10:53:17AM +0900, Kenji Rikitake wrote: > > > I finally reviewed OTP rand module change between 19.3 and 20.0. > > > While most of users won't recognize the changes, > > > the details are considerably different. > > > > > > A good news is that if you're not aware of the PRNG algorithm > > > you can keep the code untouched. The new default algorithm > > > works ok and even better than the OTP 19 with less bugs. > > > > > > On the other hand, if you explicitly specify the algorithm in rand:seed/1, > > > you are suggested to use a newly-available algorithms > > > described in the rand module manual of OTP 20.0. Actually > > > all the old (OTP 18 and 19) algorithms are now deprecated. > > > See rand module manual of OTP 20. Use exrop (default of OTP 20) in most > > > cases; > > > if you use exs1024 that should be changed to exs1024s; > > > if you stick with exsplus (default until OTP 19), use exsp instead. > > > If you use other algorithms, consider converting to exrop. > > > > > > Also, if your code depends on the output range of > > > rand:uniform/1 or rand:uniform_s/1, it has been changed as follows: > > > OTP 18 and 19: 0.0 < X < 1.0 > > > OTP 20: 0.0 =< X < 1.0 (note the =< operator) > > > where X is the output of the functions. > > > In short, since OTP 20, the functions may output 0.0. > > > > > > I noticed the changes shortly after OTP 20.0 release. > > > These changes were in the last minute (19-MAR-2017 - 26-APR-2017) > > > just before the OTP 20 release > > > and after my Erlang and Elixir Factory 2017 trip so I didn't notice. > > > I was not notified at all by Raimo Niskanen and other contributors > > > about these changed either, > > > so I had to take time and chance to review the code again. > > > I know OTP is open source software and the OTP Team can modify > > > the code for better features and removing bugs without notifying to anyone, > > > so nobody is to blame on these changes. > > > I'm assured that Raimo and the other contributors have done a great job > > > on changing the details while maintaining the compatibility > > > and fixing the trivial bugs which I left in the code. > > > > > > I would also like to note that crypto module also utilizes > > > the rand module API to simplify access to cryptographically strong > > > random number module since OTP 20. See crypto:rand_seed/{0,1} > > > and crypto:rand_uniform/2. This is a good example to make use of > > > rand module's extensible plugins. > > > > > > Thanks to Tuncer Ayaz for giving me a chance to review this. > > > > > > And always remember: if you are still dependent on random module, > > > migrate to rand module now. > > > > > > Regards, > > > Kenji Rikitake > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > -- > > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From codewiget95@REDACTED Tue Aug 29 15:26:19 2017 From: codewiget95@REDACTED (code wiget) Date: Tue, 29 Aug 2017 09:26:19 -0400 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? Message-ID: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> Hello, I have been looking on CVE at security vulnerabilities for Erlang here : https://www.cvedetails.com/vulnerability-list/vendor_id-9446/Erlang.html to assess the risks posed to Erlang servers. Based on the information on these sites, it seems that Erlang?s OTP 19+ is very ?secure? and isn?t vulnerable to any buffer overflow/stack smashing/heap smashing attacks. Would you feel comfortable leaving a open SSL port using no peer certificates on the public internet? For example, using a gen_server, do you think it is possible to handle all calls/casts/info?s properly without posing a risk to your system? Is there anything you would do special when your system was open to the public internet? -------------- next part -------------- An HTML attachment was scrubbed... URL: From silviu.cpp@REDACTED Tue Aug 29 19:34:33 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Tue, 29 Aug 2017 20:34:33 +0300 Subject: [erlang-questions] optimal way to append an element in a list inside a map Message-ID: Hello, Having a map where the value of each element it's a list : #{ 1 => [], 2 => [], ... n => []} and you need to append elements in the list for a specific key, what's the most optimal way to do this without copying the lists and the map inside the VM lot of times ? Anything better than the following solution: append_element(Key, Value, Map) -> case maps:find(Key, Map) of {ok, V} -> maps:put(Key, [Value | V], Map); _ -> maps:put(Key, [Value], Map) end. Kind regards, Silviu -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Tue Aug 29 20:11:20 2017 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 29 Aug 2017 20:11:20 +0200 Subject: [erlang-questions] Improved IO polling in erts Message-ID: Hello everyone, I've just opened up a new PR at github containing a solution to the scalability problems that the current polling mechanism in erts has. You can read more about it in the PR here: https://github.com/erlang/otp/pull/1552. Our internal measurements have shown great potential, but creating realistic benchmarks is very hard. I would therefore encourage as many as possible to test and see how the changes proposed effects your applications. Please come back with any findings/questions, be they good or bad. As this is an essential part of many applications, we want to make the new implementation as good as can be. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Tue Aug 29 20:19:54 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 29 Aug 2017 21:19:54 +0300 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: Hello, Premature optimisation is an evil ;-) I would use the following syntax: ``` append(Key, Value, Map) -> List = case Map of #{Key := Tail} -> [Value | Tail]; _ -> [Value] end, Map#{Key => List}. ``` Lists are not copied they are referenced. Maps? Hmm, I am not sure. I hope the implementation is smart enough to keep reference as well. - Dmitry > On 29 Aug 2017, at 20.34, Caragea Silviu wrote: > > Hello, > > Having a map where the value of each element it's a list : > > #{ 1 => [], 2 => [], ... n => []} > > and you need to append elements in the list for a specific key, what's the most optimal way to do this without copying the lists and the map inside the VM lot of times ? > > Anything better than the following solution: > > append_element(Key, Value, Map) -> > case maps:find(Key, Map) of > {ok, V} -> > maps:put(Key, [Value | V], Map); > _ -> > maps:put(Key, [Value], Map) > end. > > Kind regards, > Silviu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From dmytro.lytovchenko@REDACTED Tue Aug 29 20:34:41 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Tue, 29 Aug 2017 20:34:41 +0200 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: 2017-08-29 20:19 GMT+02:00 Dmitry Kolesnikov : > Hello, > > Premature optimisation is an evil ;-) > > I would use the following syntax: > ``` > append(Key, Value, Map) -> > List = case Map of > #{Key := Tail} -> > [Value | Tail]; > _ -> > [Value] > end, > Map#{Key => List}. > ``` > > Lists are not copied they are referenced. Maps? Hmm, I am not sure. I hope > the implementation is smart enough to keep reference as well. > In BEAM any object which does not fit into a machine register (such as a list or a map) will be placed on heap and a pointer will be used instead. But prepending to a list will change the list pointer value (the pointer to list head will change, the remaining tail elements will not move and will not change). This requires writing the new head into the map value. And this will incur a map update, which most likely will rebuild the map. I'm almost sure that the optimization possibilities for this are very limited and are similar to tuple optimizations (i.e. works only at creation time). > > - Dmitry > > > > On 29 Aug 2017, at 20.34, Caragea Silviu wrote: > > > > Hello, > > > > Having a map where the value of each element it's a list : > > > > #{ 1 => [], 2 => [], ... n => []} > > > > and you need to append elements in the list for a specific key, what's > the most optimal way to do this without copying the lists and the map > inside the VM lot of times ? > > > > Anything better than the following solution: > > > > append_element(Key, Value, Map) -> > > case maps:find(Key, Map) of > > {ok, V} -> > > maps:put(Key, [Value | V], Map); > > _ -> > > maps:put(Key, [Value], Map) > > end. > > > > Kind regards, > > Silviu > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbrujohalcon@REDACTED Tue Aug 29 20:29:48 2017 From: elbrujohalcon@REDACTED (Brujo Benavides) Date: Tue, 29 Aug 2017 15:29:48 -0300 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: <6D372115-FA8E-45D8-9F03-67575BB2011E@gmail.com> I think this might be what you are looking for: append(Key, Value, Map) -> maps:update_with(Key, fun(Tail) -> [Value | Tail] end, [], Map). Brujo Benavides > On Aug 29, 2017, at 15:19, Dmitry Kolesnikov wrote: > > Hello, > > Premature optimisation is an evil ;-) > > I would use the following syntax: > ``` > append(Key, Value, Map) -> > List = case Map of > #{Key := Tail} -> > [Value | Tail]; > _ -> > [Value] > end, > Map#{Key => List}. > ``` > > Lists are not copied they are referenced. Maps? Hmm, I am not sure. I hope the implementation is smart enough to keep reference as well. > > - Dmitry > > >> On 29 Aug 2017, at 20.34, Caragea Silviu wrote: >> >> Hello, >> >> Having a map where the value of each element it's a list : >> >> #{ 1 => [], 2 => [], ... n => []} >> >> and you need to append elements in the list for a specific key, what's the most optimal way to do this without copying the lists and the map inside the VM lot of times ? >> >> Anything better than the following solution: >> >> append_element(Key, Value, Map) -> >> case maps:find(Key, Map) of >> {ok, V} -> >> maps:put(Key, [Value | V], Map); >> _ -> >> maps:put(Key, [Value], Map) >> end. >> >> Kind regards, >> Silviu >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From silviu.cpp@REDACTED Tue Aug 29 20:41:35 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Tue, 29 Aug 2017 21:41:35 +0300 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: Hmm, Basically what you are saying is that any update over a map requires rebuilding of the entire map ? I doubt as time implementation seems so lame.. Any way that we con confirm or not this theory ? Silviu On Tue, Aug 29, 2017 at 9:34 PM, Dmytro Lytovchenko < dmytro.lytovchenko@REDACTED> wrote: > > > 2017-08-29 20:19 GMT+02:00 Dmitry Kolesnikov : > >> Hello, >> >> Premature optimisation is an evil ;-) >> >> I would use the following syntax: >> ``` >> append(Key, Value, Map) -> >> List = case Map of >> #{Key := Tail} -> >> [Value | Tail]; >> _ -> >> [Value] >> end, >> Map#{Key => List}. >> ``` >> >> Lists are not copied they are referenced. Maps? Hmm, I am not sure. I >> hope the implementation is smart enough to keep reference as well. >> > > In BEAM any object which does not fit into a machine register (such as a > list or a map) will be placed on heap and a pointer will be used instead. > But prepending to a list will change the list pointer value (the pointer to > list head will change, the remaining tail elements will not move and will > not change). This requires writing the new head into the map value. And > this will incur a map update, which most likely will rebuild the map. I'm > almost sure that the optimization possibilities for this are very limited > and are similar to tuple optimizations (i.e. works only at creation time). > > >> >> - Dmitry >> >> >> > On 29 Aug 2017, at 20.34, Caragea Silviu wrote: >> > >> > Hello, >> > >> > Having a map where the value of each element it's a list : >> > >> > #{ 1 => [], 2 => [], ... n => []} >> > >> > and you need to append elements in the list for a specific key, what's >> the most optimal way to do this without copying the lists and the map >> inside the VM lot of times ? >> > >> > Anything better than the following solution: >> > >> > append_element(Key, Value, Map) -> >> > case maps:find(Key, Map) of >> > {ok, V} -> >> > maps:put(Key, [Value | V], Map); >> > _ -> >> > maps:put(Key, [Value], Map) >> > end. >> > >> > Kind regards, >> > Silviu >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Aug 29 20:45:21 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 29 Aug 2017 21:45:21 +0300 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> Message-ID: yes, it is reliable. there are some performance issues, but they are complicated. feel free to do it -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Tue Aug 29 20:56:28 2017 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 29 Aug 2017 20:56:28 +0200 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: On Tue, Aug 29, 2017 at 8:41 PM, Caragea Silviu wrote: > Hmm, > Basically what you are saying is that any update over a map requires > rebuilding of the entire map ? I doubt as time implementation seems so > lame.. > Well, Erlang is a functional language and as such, you can't update any data structure in place. Updates create a new data structure and others may still have references to the original. Always. For lists, you can reuse the original list as the tail. For tuples, you have to copy the top level. For maps, it depends: the implementation is a HAMT structure, which means that it depends on the contents how much will be rewritten when updating; I don't know what the worst case is. Whatever the case, there will be a new map data structure created. regards, Vlad > Any way that we con confirm or not this theory ? > > Silviu > > On Tue, Aug 29, 2017 at 9:34 PM, Dmytro Lytovchenko < > dmytro.lytovchenko@REDACTED> wrote: > >> >> >> 2017-08-29 20:19 GMT+02:00 Dmitry Kolesnikov : >> >>> Hello, >>> >>> Premature optimisation is an evil ;-) >>> >>> I would use the following syntax: >>> ``` >>> append(Key, Value, Map) -> >>> List = case Map of >>> #{Key := Tail} -> >>> [Value | Tail]; >>> _ -> >>> [Value] >>> end, >>> Map#{Key => List}. >>> ``` >>> >>> Lists are not copied they are referenced. Maps? Hmm, I am not sure. I >>> hope the implementation is smart enough to keep reference as well. >>> >> >> In BEAM any object which does not fit into a machine register (such as a >> list or a map) will be placed on heap and a pointer will be used instead. >> But prepending to a list will change the list pointer value (the pointer to >> list head will change, the remaining tail elements will not move and will >> not change). This requires writing the new head into the map value. And >> this will incur a map update, which most likely will rebuild the map. I'm >> almost sure that the optimization possibilities for this are very limited >> and are similar to tuple optimizations (i.e. works only at creation time). >> >> >>> >>> - Dmitry >>> >>> >>> > On 29 Aug 2017, at 20.34, Caragea Silviu wrote: >>> > >>> > Hello, >>> > >>> > Having a map where the value of each element it's a list : >>> > >>> > #{ 1 => [], 2 => [], ... n => []} >>> > >>> > and you need to append elements in the list for a specific key, what's >>> the most optimal way to do this without copying the lists and the map >>> inside the VM lot of times ? >>> > >>> > Anything better than the following solution: >>> > >>> > append_element(Key, Value, Map) -> >>> > case maps:find(Key, Map) of >>> > {ok, V} -> >>> > maps:put(Key, [Value | V], Map); >>> > _ -> >>> > maps:put(Key, [Value], Map) >>> > end. >>> > >>> > Kind regards, >>> > Silviu >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Aug 29 21:00:11 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 29 Aug 2017 22:00:11 +0300 Subject: [erlang-questions] Improved IO polling in erts In-Reply-To: References: Message-ID: hi. perhaps it will help us to capture 1 gbit of udp without failure On Tue, Aug 29, 2017 at 9:11 PM, Lukas Larsson wrote: > Hello everyone, > > I've just opened up a new PR at github containing a solution to the > scalability problems that the current polling mechanism in erts has. You > can read more about it in the PR here: https://github.com/ > erlang/otp/pull/1552. > > Our internal measurements have shown great potential, but creating > realistic benchmarks is very hard. I would therefore encourage as many as > possible to test and see how the changes proposed effects your > applications. Please come back with any findings/questions, be they good or > bad. As this is an essential part of many applications, we want to make the > new implementation as good as can be. > > Lukas > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Tue Aug 29 21:01:16 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Tue, 29 Aug 2017 21:01:16 +0200 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: Maps have two implementations depending on number of keys. Small maps (<= 32 keys) are basically a key-tuple and a value tuple. Updating the value of an existing key will copy the value-tuple while reusing the key-tuple. Large maps (> 32 keys) are basically a tree (HAMT). All nodes from the root to the one containing the key must be copied while all other nodes can be reused. /Sverker On 08/29/2017 08:41 PM, Caragea Silviu wrote: > Hmm, > > Basically what you are saying is that any update over a map requires > rebuilding of the entire map ? I doubt as time implementation seems so > lame.. > > Any way that we con confirm or not this theory ? > > Silviu > > On Tue, Aug 29, 2017 at 9:34 PM, Dmytro Lytovchenko < > dmytro.lytovchenko@REDACTED> wrote: > >> >> 2017-08-29 20:19 GMT+02:00 Dmitry Kolesnikov : >> >>> Hello, >>> >>> Premature optimisation is an evil ;-) >>> >>> I would use the following syntax: >>> ``` >>> append(Key, Value, Map) -> >>> List = case Map of >>> #{Key := Tail} -> >>> [Value | Tail]; >>> _ -> >>> [Value] >>> end, >>> Map#{Key => List}. >>> ``` >>> >>> Lists are not copied they are referenced. Maps? Hmm, I am not sure. I >>> hope the implementation is smart enough to keep reference as well. >>> >> In BEAM any object which does not fit into a machine register (such as a >> list or a map) will be placed on heap and a pointer will be used instead. >> But prepending to a list will change the list pointer value (the pointer to >> list head will change, the remaining tail elements will not move and will >> not change). This requires writing the new head into the map value. And >> this will incur a map update, which most likely will rebuild the map. I'm >> almost sure that the optimization possibilities for this are very limited >> and are similar to tuple optimizations (i.e. works only at creation time). >> >> >>> - Dmitry >>> >>> >>>> On 29 Aug 2017, at 20.34, Caragea Silviu wrote: >>>> >>>> Hello, >>>> >>>> Having a map where the value of each element it's a list : >>>> >>>> #{ 1 => [], 2 => [], ... n => []} >>>> >>>> and you need to append elements in the list for a specific key, what's >>> the most optimal way to do this without copying the lists and the map >>> inside the VM lot of times ? >>>> Anything better than the following solution: >>>> >>>> append_element(Key, Value, Map) -> >>>> case maps:find(Key, Map) of >>>> {ok, V} -> >>>> maps:put(Key, [Value | V], Map); >>>> _ -> >>>> maps:put(Key, [Value], Map) >>>> end. >>>> >>>> Kind regards, >>>> Silviu >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Tue Aug 29 21:02:51 2017 From: codewiget95@REDACTED (code wiget) Date: Tue, 29 Aug 2017 15:02:51 -0400 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> Message-ID: <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> Can you elaborate? What performance issues do I need to work around? And when you say reliable, you mean there aren?t any attacks that I should be concerned with? Thanks for the help! From max.lapshin@REDACTED Tue Aug 29 21:07:14 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 29 Aug 2017 22:07:14 +0300 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> Message-ID: take a look at session cache issues -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Tue Aug 29 21:15:05 2017 From: codewiget95@REDACTED (code wiget) Date: Tue, 29 Aug 2017 15:15:05 -0400 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> Message-ID: So that has to do with load, which is fine because this particular instance should be light load - I am speaking more towards security. If someone connects, are there attacks they can do with access to an Erlang controlled socket? From silviu.cpp@REDACTED Tue Aug 29 21:16:37 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Tue, 29 Aug 2017 22:16:37 +0300 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: Hello, Thanks for clarification. I thought that in background erlang can see when there is no other reference to the original map/list (for example in case of accumulator inside a recursive fun) and will append to the original object while updating the reference instead of making partial copy. Silviu On Tue, Aug 29, 2017 at 10:01 PM, Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > Maps have two implementations depending on number of keys. > > Small maps (<= 32 keys) are basically a key-tuple and a value tuple. > Updating the value of an existing key will copy the value-tuple > while reusing the key-tuple. > > Large maps (> 32 keys) are basically a tree (HAMT). > All nodes from the root to the one containing the key must be copied > while all other nodes can be reused. > > /Sverker > > On 08/29/2017 08:41 PM, Caragea Silviu wrote: > > Hmm, > > Basically what you are saying is that any update over a map requires > rebuilding of the entire map ? I doubt as time implementation seems so > lame.. > > Any way that we con confirm or not this theory ? > > Silviu > > On Tue, Aug 29, 2017 at 9:34 PM, Dmytro Lytovchenko wrote: > > > > 2017-08-29 20:19 GMT+02:00 Dmitry Kolesnikov : > > > Hello, > > Premature optimisation is an evil ;-) > > I would use the following syntax: > ``` > append(Key, Value, Map) -> > List = case Map of > #{Key := Tail} -> > [Value | Tail]; > _ -> > [Value] > end, > Map#{Key => List}. > ``` > > Lists are not copied they are referenced. Maps? Hmm, I am not sure. I > hope the implementation is smart enough to keep reference as well. > > > In BEAM any object which does not fit into a machine register (such as a > list or a map) will be placed on heap and a pointer will be used instead. > But prepending to a list will change the list pointer value (the pointer to > list head will change, the remaining tail elements will not move and will > not change). This requires writing the new head into the map value. And > this will incur a map update, which most likely will rebuild the map. I'm > almost sure that the optimization possibilities for this are very limited > and are similar to tuple optimizations (i.e. works only at creation time). > > > > - Dmitry > > > > On 29 Aug 2017, at 20.34, Caragea Silviu wrote: > > Hello, > > Having a map where the value of each element it's a list : > > #{ 1 => [], 2 => [], ... n => []} > > and you need to append elements in the list for a specific key, what's > > the most optimal way to do this without copying the lists and the map > inside the VM lot of times ? > > Anything better than the following solution: > > append_element(Key, Value, Map) -> > case maps:find(Key, Map) of > {ok, V} -> > maps:put(Key, [Value | V], Map); > _ -> > maps:put(Key, [Value], Map) > end. > > Kind regards, > Silviu > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Aug 29 21:34:40 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 29 Aug 2017 19:34:40 +0000 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: If you measure and find the map to be too slow for you, the use case you have where the contents are ephemeral suggest you should look into using an ETS table of type bag or ordered_bag. You can get some really fast code by using this if you don't need the persistence the map provides. I'd measure first though. Chances are the maps are plenty fast. You can always encapsulate the implementation so you can subst in ETS later if need be. On Tue, Aug 29, 2017 at 9:16 PM Caragea Silviu wrote: > Hello, > > Thanks for clarification. I thought that in background erlang can see when > there is no other reference to the original map/list (for example in case > of accumulator inside a recursive fun) and will append to the original > object while updating the reference instead of making partial copy. > > Silviu > > On Tue, Aug 29, 2017 at 10:01 PM, Sverker Eriksson < > sverker.eriksson@REDACTED> wrote: > >> Maps have two implementations depending on number of keys. >> >> Small maps (<= 32 keys) are basically a key-tuple and a value tuple. >> Updating the value of an existing key will copy the value-tuple >> while reusing the key-tuple. >> >> Large maps (> 32 keys) are basically a tree (HAMT). >> All nodes from the root to the one containing the key must be copied >> while all other nodes can be reused. >> >> /Sverker >> >> On 08/29/2017 08:41 PM, Caragea Silviu wrote: >> >> Hmm, >> >> Basically what you are saying is that any update over a map requires >> rebuilding of the entire map ? I doubt as time implementation seems so >> lame.. >> >> Any way that we con confirm or not this theory ? >> >> Silviu >> >> On Tue, Aug 29, 2017 at 9:34 PM, Dmytro Lytovchenko wrote: >> >> >> >> 2017-08-29 20:19 GMT+02:00 Dmitry Kolesnikov : >> >> >> Hello, >> >> Premature optimisation is an evil ;-) >> >> I would use the following syntax: >> ``` >> append(Key, Value, Map) -> >> List = case Map of >> #{Key := Tail} -> >> [Value | Tail]; >> _ -> >> [Value] >> end, >> Map#{Key => List}. >> ``` >> >> Lists are not copied they are referenced. Maps? Hmm, I am not sure. I >> hope the implementation is smart enough to keep reference as well. >> >> >> In BEAM any object which does not fit into a machine register (such as a >> list or a map) will be placed on heap and a pointer will be used instead. >> But prepending to a list will change the list pointer value (the pointer to >> list head will change, the remaining tail elements will not move and will >> not change). This requires writing the new head into the map value. And >> this will incur a map update, which most likely will rebuild the map. I'm >> almost sure that the optimization possibilities for this are very limited >> and are similar to tuple optimizations (i.e. works only at creation time). >> >> >> >> - Dmitry >> >> >> >> On 29 Aug 2017, at 20.34, Caragea Silviu wrote: >> >> Hello, >> >> Having a map where the value of each element it's a list : >> >> #{ 1 => [], 2 => [], ... n => []} >> >> and you need to append elements in the list for a specific key, what's >> >> the most optimal way to do this without copying the lists and the map >> inside the VM lot of times ? >> >> Anything better than the following solution: >> >> append_element(Key, Value, Map) -> >> case maps:find(Key, Map) of >> {ok, V} -> >> maps:put(Key, [Value | V], Map); >> _ -> >> maps:put(Key, [Value], Map) >> end. >> >> Kind regards, >> Silviu >> _______________________________________________ >> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions >> >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott_ribe@REDACTED Tue Aug 29 20:44:01 2017 From: scott_ribe@REDACTED (scott ribe) Date: Tue, 29 Aug 2017 12:44:01 -0600 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: > On Aug 29, 2017, at 12:34 PM, Dmytro Lytovchenko wrote: > > > > 2017-08-29 20:19 GMT+02:00 Dmitry Kolesnikov : > Hello, > > Premature optimisation is an evil ;-) > > I would use the following syntax: > ``` > append(Key, Value, Map) -> > List = case Map of > #{Key := Tail} -> > [Value | Tail]; > _ -> > [Value] > end, > Map#{Key => List}. > ``` > > Lists are not copied they are referenced. Maps? Hmm, I am not sure. I hope the implementation is smart enough to keep reference as well. > > In BEAM any object which does not fit into a machine register (such as a list or a map) will be placed on heap and a pointer will be used instead. But prepending to a list will change the list pointer value (the pointer to list head will change, the remaining tail elements will not move and will not change). This requires writing the new head into the map value. And this will incur a map update, which most likely will rebuild the map. I'm almost sure that the optimization possibilities for this are very limited and are similar to tuple optimizations (i.e. works only at creation time). Actually, it's perfectly possible to optimize the map insertion in a way analogous to the list prepending. A new root will be created, sharing old structure. If you think of some tree, then in fact new nodes are created from the leaf value you just added back up to the root, so not as cheap as list prepending, but way cheaper than copying the tree. -- Scott Ribe scott_ribe@REDACTED (303) 722-0567 From z@REDACTED Tue Aug 29 23:28:55 2017 From: z@REDACTED (Danil Zagoskin) Date: Wed, 30 Aug 2017 00:28:55 +0300 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> Message-ID: Why ask? Deploy your server and use security analysis tool like https://www.ssllabs.com/ssltest/index.html We get A rating (which means there are no known vulnerabilities) by disabling SSLv3 and some weak cipher suites (the tool above will tell you what to disable). On Tue, Aug 29, 2017 at 10:15 PM, code wiget wrote: > So that has to do with load, which is fine because this particular > instance should be light load - I am speaking more towards security. If > someone connects, are there attacks they can do with access to an Erlang > controlled socket? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Aug 30 00:20:02 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 29 Aug 2017 18:20:02 -0400 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> Message-ID: <20170829221959.GA51868@ferdmbp.local> On 08/29, code wiget wrote: >So that has to do with load, which is fine because this particular instance should be light load - I am speaking more towards security. If someone connects, are there attacks they can do with access to an Erlang controlled socket? > Aside from the cache issues Max has mentioned, there's a few configuration values you might want by default: [{ciphers, CipherList}, % see below {honor_cipher_order, true}, % pick the server-defined order of ciphers {secure_renegotiate, true}, % prevent renegotiation hijacks {client_renegotiation, false}, % prevent clients DoSing w/ renegs {versions, ['tlsv1.2', 'tlsv1.1']}, % add tlsv1 if you must {reuse_sessions, false}, % drop session cache for perf {ecc, EllipticCurves} % see below ]. A safe CipherList can be those enumerated in https://github.com/heroku/snit/blob/master/src/snit.app.src#L45-L83 for example, though the format in that config is meant to contain both the OpenSSL-readable format and the Erlang-accepted one. The order of elliptic curves I like is the one at https://github.com/heroku/snit/blob/master/src/snit.app.src#L116-L121 -- it is not the strongest, but aligns with what AWS ELBs prefer (secp256r1 first) which gives a decent compromise between performance and safety. Stronger curves at 512b roughly double the time a handshake takes, but if you prefer the safety to the perf, reorder them to be first. Furthermore, the following values can go in your sys.config file to further modify the SSL behaviour: {ssl, [ {bypass_pem_cache, true}, % bypass PEM cache (see below) {session_cb, ssl_cache_null}, % see below {session_cb_init_args, []} % (cont) ]} The PEM cache is a cache used whenever you have disk-based certificates. In cases where you use in-memory certificates, it can act as a bottleneck. See https://blog.heroku.com/how-we-sped-up-sni-tls-handshakes-by-5x for my writeup on the topic. The last one about the session callback is a further cache that you may disable if you hit performance issues. It uses the callback at http://erlang.org/doc/man/ssl_session_cache_api.html to configure how to store session data. A gotcha is that this table still sees some use even if you disable the session cache (or at least it did last time I looked). As such, you can provide an empty module like the following one to fully bypass it: -module(ssl_cache_null). -behaviour(ssl_session_cache_api). -export([init/1, terminate/1, lookup/2, update/3, delete/2, foldl/3, select_session/2, size/1]). init(_) -> disabled. terminate(_) -> disabled. lookup(_,_) -> undefined. update(_,_,_) -> disabled. delete(_,_) -> disabled. foldl(_,Acc,_) -> Acc. select_session(_,_) -> []. size(_) -> 0. With this module part of your project along with the config above, you should get quite decent performance with it. Back in the days I was at heroku, we went close to what Amazon ELBs could do in terms of performance. Maybe a few milliseconds slower on average, but nearly an order of magnitude faster on 99th percentiles. From mononcqc@REDACTED Wed Aug 30 00:23:55 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 29 Aug 2017 18:23:55 -0400 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: <20170829221959.GA51868@ferdmbp.local> References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> <20170829221959.GA51868@ferdmbp.local> Message-ID: <20170829222354.GB51868@ferdmbp.local> On 08/29, Fred Hebert wrote: >Aside from the cache issues Max has mentioned, there's a few >configuration values you might want by default: > > [{ciphers, CipherList}, % see below > {honor_cipher_order, true}, % pick the server-defined order of ciphers > {secure_renegotiate, true}, % prevent renegotiation hijacks > {client_renegotiation, false}, % prevent clients DoSing w/ renegs > {versions, ['tlsv1.2', 'tlsv1.1']}, % add tlsv1 if you must > {reuse_sessions, false}, % drop session cache for perf > {ecc, EllipticCurves} % see below > ]. > Forgot to add {honor_ecc_order, true} to that list if you use the ecc option! From ok@REDACTED Wed Aug 30 01:44:57 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 30 Aug 2017 11:44:57 +1200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170829083528.GA91917@erix.ericsson.se> References: <20170829083528.GA91917@erix.ericsson.se> Message-ID: On 29/08/17 8:35 PM, Raimo Niskanen wrote: > > Regarding the changed uniform float behaviour: it is the functions > rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously > (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now > (OTP-20.0) documented to return 0.0 =< X < 1.0. There are applications of random numbers for which it is important that 0 never be returned. Of course, nothing stops me writing uniform_nonzero() -> X = rand:uniform(), if X > 0.0 -> X ; X =< 0.0 -> uniform_nonzero() end. but it would be nice to have this already in the library. (I know the old library had the same gap. But with such a major reworking of random number generation, it's time to close it.) From ok@REDACTED Wed Aug 30 02:04:00 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 30 Aug 2017 12:04:00 +1200 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: On 30/08/17 5:34 AM, Caragea Silviu wrote: > Having a map where the value of each element is a list : > > #{ 1 => [], 2 => [], ... n => []} > > and you need to append elements in the list for a specific key, You write "append", and to append an element X to a list L is to compute L ++ [X] somehow. But your sample code computes [X] ++ L (as [X|L]), which already does not copy L, and is not appending but prepending. case Map of #{ Key := List } -> Map#{Key := [Value|List]} ; _ -> Map#{Key => [Value]} end does what you appear to want in native syntax. (Tested.) What performance difference there may be between this and your example code is a quality-of-implementation issue discoverable by benchmarking. To repeat, it doesn't copy List. But it has to copy Map in order to change it. Parts of the new Map will share structure with the old one, but how much and how much of a cost the copying is will be a quality-of-implementation issue and depends on the number of keys in the Map. It would help to know more about the problem you are actually trying to solve. BTW "most optimal" is redundant; "optimal" already means "best". From ok@REDACTED Wed Aug 30 02:18:55 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 30 Aug 2017 12:18:55 +1200 Subject: [erlang-questions] optimal way to append an element in a list inside a map In-Reply-To: References: Message-ID: On 30/08/17 6:41 AM, Caragea Silviu wrote: > Hmm, > > Basically what you are saying is that any update over a map requires > rebuilding of the entire map ? It doesn't take much poking around on the Web before you discover that Erlang maps use "a HAMT (Hash-Array Mapped Trie) internally once the map grows enough. It is a clever data structure which combines the properties of a hash table with a (level compressed) trie to provide fast lookup as well as persistence. This is the same data structure languages like Clojure, Scala and Haskell (unordered-containers) use. They were designed by the now late Phil Bagwell and the Erlang variant leans itself up against the work of Rich Hickey in Clojure." -- Jesper L. Andersen, at https://medium.com/@jlouis666/breaking-erlang-maps-1-31952b8729e6 Large maps in Erlang are therefore trees with fairly wide nodes, but updating a map WON'T copy the whole thing. From frank.muller.erl@REDACTED Wed Aug 30 06:28:38 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 30 Aug 2017 04:28:38 +0000 Subject: [erlang-questions] Flush "stdout" buffer Message-ID: Hi guys, I'd like to know if there's an equivalent function to C fflush(stdout) ? I've a small Erlang program which prints data to stdout. After every io:format call, I'd like to ensure output is flushed to screen. Thanks in adavance. /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Wed Aug 30 08:42:02 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 30 Aug 2017 08:42:02 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: <20170829083528.GA91917@erix.ericsson.se> Message-ID: <20170830064202.GA17946@erix.ericsson.se> On Wed, Aug 30, 2017 at 11:44:57AM +1200, Richard A. O'Keefe wrote: > > > On 29/08/17 8:35 PM, Raimo Niskanen wrote: > > > > Regarding the changed uniform float behaviour: it is the functions > > rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously > > (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now > > (OTP-20.0) documented to return 0.0 =< X < 1.0. > > There are applications of random numbers for which it is important > that 0 never be returned. Of course, nothing stops me writing What kind of applications? I would like to get a grip on how needed this function is? > > uniform_nonzero() -> > X = rand:uniform(), > if X > 0.0 -> X > ; X =< 0.0 -> uniform_nonzero() > end. > > but it would be nice to have this already in the library. > > (I know the old library had the same gap. But with such a major > reworking of random number generation, it's time to close it.) We chose 0.0 =< X < 1.0 because it was most like the integer generator i.e including the lower bound but excluding the upper. And as you say it is easy to exclude the lower bound. If we would implement 0.0 < X < 1.0, would then 0.0 =< X =< 1.0 also be missing, and for completeness 0.0 < X =< 1.0? Which of the four are worth implementing? You clould argue that including both bounds is the most general because it is easy to retry if you want to exclude a value. Maybe something like: uniform(Opts) -> float() uniform_s(Opts, State) -> float() Opts :: [(exclude_zero | exclude_one)] > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From zxq9@REDACTED Wed Aug 30 08:48:16 2017 From: zxq9@REDACTED (zxq9) Date: Wed, 30 Aug 2017 15:48:16 +0900 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170830064202.GA17946@erix.ericsson.se> References: <20170830064202.GA17946@erix.ericsson.se> Message-ID: <1963000.erbGFRGZg1@changa> On 2017?08?30? ??? 08:42:02 Raimo Niskanen wrote: > On Wed, Aug 30, 2017 at 11:44:57AM +1200, Richard A. O'Keefe wrote: > > > > > > On 29/08/17 8:35 PM, Raimo Niskanen wrote: > > > > > > Regarding the changed uniform float behaviour: it is the functions > > > rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously > > > (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now > > > (OTP-20.0) documented to return 0.0 =< X < 1.0. > > > > There are applications of random numbers for which it is important > > that 0 never be returned. Of course, nothing stops me writing > > What kind of applications? I would like to get a grip on how needed this > function is? Any function where a zero would propagate. This can be exactly as bad as accidentally comparing a NULL in SQL. -Craig From raimo+erlang-questions@REDACTED Wed Aug 30 08:54:30 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 30 Aug 2017 08:54:30 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <1963000.erbGFRGZg1@changa> References: <20170830064202.GA17946@erix.ericsson.se> <1963000.erbGFRGZg1@changa> Message-ID: <20170830065430.GB17946@erix.ericsson.se> On Wed, Aug 30, 2017 at 03:48:16PM +0900, zxq9 wrote: > On 2017?08?30? ??? 08:42:02 Raimo Niskanen wrote: > > On Wed, Aug 30, 2017 at 11:44:57AM +1200, Richard A. O'Keefe wrote: > > > > > > > > > On 29/08/17 8:35 PM, Raimo Niskanen wrote: > > > > > > > > Regarding the changed uniform float behaviour: it is the functions > > > > rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously > > > > (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now > > > > (OTP-20.0) documented to return 0.0 =< X < 1.0. > > > > > > There are applications of random numbers for which it is important > > > that 0 never be returned. Of course, nothing stops me writing > > > > What kind of applications? I would like to get a grip on how needed this > > function is? > > Any function where a zero would propagate. > > This can be exactly as bad as accidentally comparing a NULL in SQL. That's vague for me. Are you saying it is a common enought use pattern to divide with a random number? Are there other reasons when a float() =:= 0.0 is fatal? > > -Craig -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From zxq9@REDACTED Wed Aug 30 09:14:56 2017 From: zxq9@REDACTED (zxq9) Date: Wed, 30 Aug 2017 16:14:56 +0900 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170830065430.GB17946@erix.ericsson.se> References: <1963000.erbGFRGZg1@changa> <20170830065430.GB17946@erix.ericsson.se> Message-ID: <2691824.Matf7LgGbk@changa> On 2017?08?30? ??? 08:54:30 Raimo Niskanen wrote: > On Wed, Aug 30, 2017 at 03:48:16PM +0900, zxq9 wrote: > > On 2017?08?30? ??? 08:42:02 Raimo Niskanen wrote: > > > On Wed, Aug 30, 2017 at 11:44:57AM +1200, Richard A. O'Keefe wrote: > > > > > > > > > > > > On 29/08/17 8:35 PM, Raimo Niskanen wrote: > > > > > > > > > > Regarding the changed uniform float behaviour: it is the functions > > > > > rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously > > > > > (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now > > > > > (OTP-20.0) documented to return 0.0 =< X < 1.0. > > > > > > > > There are applications of random numbers for which it is important > > > > that 0 never be returned. Of course, nothing stops me writing > > > > > > What kind of applications? I would like to get a grip on how needed this > > > function is? > > > > Any function where a zero would propagate. > > > > This can be exactly as bad as accidentally comparing a NULL in SQL. > > That's vague for me. > > Are you saying it is a common enought use pattern to divide with a > random number? Are there other reasons when a float() =:= 0.0 is fatal? It is relatively common whenever it is guaranteed to be safe! Otherwise it becomes a guarded expression. Sure, that is a case of "well, just write it so that it can't do that" -- but the original function spec told us we didn't need to do that, so there is code out there that would rely on not using a factor of 0.0. I've probably written some in game servers, actually. Propagating the product of multiplication by 0.0 is the more common problem I've seen, by the way, as opposed to division. Consider: character stat generation in games, offset-by-random-factor calculations where accidentally getting exactly the same result is catastrophic, anti-precision routines in some aiming devices and simulations, adding wiggle to character pathfinding, unstuck() type routines, mutating a value in evolutionary design algorithms, and so on. Very few of these cases are catastrophic and many would simply be applied again if the initial attempt failed, but a few can be very bad depending on how the system in which they are used is designed. The problem isn't so much that "there aren't many use cases" or "the uses aren't common" as much as the API was originally documented that way, and it has changed for no apparent reason. Zero has a very special place in mathematics and should be treated carefully. I think ROK would have objected a lot less had the original spec been 0.0 =< X =< 1.0 (which is different from being 0.0 =< X < 1.0; which is another point of potentially dangerous weirdness). I'm curious to see what examples he comes up with. The ones above are just off the top of my head, and like I mentioned most of my personal examples don't happen to be really catastrophic in most cases because many of them involve offsetting from a known value (which would be relatively safe to reuse) or situations where failures are implicitly assumed to provoke retries. -Craig From raimo+erlang-questions@REDACTED Wed Aug 30 10:29:12 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 30 Aug 2017 10:29:12 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <2691824.Matf7LgGbk@changa> References: <1963000.erbGFRGZg1@changa> <20170830065430.GB17946@erix.ericsson.se> <2691824.Matf7LgGbk@changa> Message-ID: <20170830082912.GA29138@erix.ericsson.se> On Wed, Aug 30, 2017 at 04:14:56PM +0900, zxq9 wrote: > On 2017?08?30? ??? 08:54:30 Raimo Niskanen wrote: > > On Wed, Aug 30, 2017 at 03:48:16PM +0900, zxq9 wrote: > > > On 2017?08?30? ??? 08:42:02 Raimo Niskanen wrote: > > > > On Wed, Aug 30, 2017 at 11:44:57AM +1200, Richard A. O'Keefe wrote: > > > > > > > > > > > > > > > On 29/08/17 8:35 PM, Raimo Niskanen wrote: > > > > > > > > > > > > Regarding the changed uniform float behaviour: it is the functions > > > > > > rand:uniform/0 and rand:uniform_s/1 this concerns. They were previously > > > > > > (OTP-19.3) documented to output a value 0.0 < X < 1.0 and are now > > > > > > (OTP-20.0) documented to return 0.0 =< X < 1.0. > > > > > > > > > > There are applications of random numbers for which it is important > > > > > that 0 never be returned. Of course, nothing stops me writing > > > > > > > > What kind of applications? I would like to get a grip on how needed this > > > > function is? > > > > > > Any function where a zero would propagate. > > > > > > This can be exactly as bad as accidentally comparing a NULL in SQL. > > > > That's vague for me. > > > > Are you saying it is a common enought use pattern to divide with a > > random number? Are there other reasons when a float() =:= 0.0 is fatal? > > It is relatively common whenever it is guaranteed to be safe! Otherwise it becomes a guarded expression. > > Sure, that is a case of "well, just write it so that it can't do that" -- but the original function spec told us we didn't need to do that, so there is code out there that would rely on not using a factor of 0.0. I've probably written some in game servers, actually. > > Propagating the product of multiplication by 0.0 is the more common problem I've seen, by the way, as opposed to division. > > Consider: character stat generation in games, offset-by-random-factor calculations where accidentally getting exactly the same result is catastrophic, anti-precision routines in some aiming devices and simulations, adding wiggle to character pathfinding, unstuck() type routines, mutating a value in evolutionary design algorithms, and so on. > > Very few of these cases are catastrophic and many would simply be applied again if the initial attempt failed, but a few can be very bad depending on how the system in which they are used is designed. The problem isn't so much that "there aren't many use cases" or "the uses aren't common" as much as the API was originally documented that way, and it has changed for no apparent reason. Zero has a very special place in mathematics and should be treated carefully. The spec did not match the reality. Either had to be corrected. It is in general safer to change the documentation to match the reality. So I do not agree that the spec changed for no apparent reason. Furthermore Java's Random:nextFloat(), Python's random.random() and Ruby's Random.rand all generate in the same interval: http://docs.oracle.com/javase/6/docs/api/java/util/Random.html#nextFloat() https://docs.python.org/3/library/random.html#random.random http://ruby-doc.org/core-2.0.0/Random.html#method-i-rand I think this all boils down to the fact that digital floating point values (IEEE 754) has limited precision and in the interval 0.0 to 1.0 are better regarded as 53 bit fixed point values. A half open interval matches integer random number generators that also in general use half open intervals. With half open intervals you can generate numbers in [0.0,1.0) and other numbers in [1.0,2.0), where the number 1.0 belongs to only one of these intervals. This I think is a good default behaviour. > > I think ROK would have objected a lot less had the original spec been 0.0 =< X =< 1.0 (which is different from being 0.0 =< X < 1.0; which is another point of potentially dangerous weirdness). I'm curious to see what examples he comes up with. The ones above are just off the top of my head, and like I mentioned most of my personal examples don't happen to be really catastrophic in most cases because many of them involve offsetting from a known value (which would be relatively safe to reuse) or situations where failures are implicitly assumed to provoke retries. > > -Craig -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From zxq9@REDACTED Wed Aug 30 12:33:04 2017 From: zxq9@REDACTED (zxq9) Date: Wed, 30 Aug 2017 19:33:04 +0900 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170830082912.GA29138@erix.ericsson.se> References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> Message-ID: <13223505.cVJpAap13P@changa> On 2017?08?30? ??? 10:29:12 Raimo Niskanen wrote: > It is in general safer to change the documentation to match the reality. Wow. From ulf@REDACTED Wed Aug 30 13:31:48 2017 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 30 Aug 2017 13:31:48 +0200 Subject: [erlang-questions] lists:pmap ? In-Reply-To: References: Message-ID: I discussed the problems with a general pmap in a presentation back in 2009. https://www.infoq.com/presentations/wiger-multicore-erlang The purpose wasn't really to argue against a general pmap, but rather to use it as a simple example to illustrate the problems with treating erlang processes as 'fibers' rather than independent actors. BR, Ulf W 2017-08-22 10:42 GMT+02:00 Ola Andersson A : > Reading the discussion about list:mapfind reminded me of a function I > rediscovered recently. > It's rpc:pmap/3 that was originally defined for use in a distributed > environment, spreading out the processing over several nodes. It actually > works on a single multicore node as well even though it wasn't designed for > that purpose. > With the limited tests I have done it seems to significantly outperform > lists:map/2 and also scale reasonably well. The almost negligible cost of > spawning erlang processes is still amazing. > How about adding a lists:pmap/2 function, designed for multicore, in the > lists module? > /OLA. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Wed Aug 30 16:28:04 2017 From: codewiget95@REDACTED (code wiget) Date: Wed, 30 Aug 2017 10:28:04 -0400 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: <20170829222354.GB51868@ferdmbp.local> References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> <20170829221959.GA51868@ferdmbp.local> <20170829222354.GB51868@ferdmbp.local> Message-ID: Fred, This is exactly what I needed, thank you. This will serve as a great reference manual. > On Aug 29, 2017, at 6:23 PM, Fred Hebert wrote: > > On 08/29, Fred Hebert wrote: >> Aside from the cache issues Max has mentioned, there's a few configuration values you might want by default: >> >> [{ciphers, CipherList}, % see below >> {honor_cipher_order, true}, % pick the server-defined order of ciphers >> {secure_renegotiate, true}, % prevent renegotiation hijacks >> {client_renegotiation, false}, % prevent clients DoSing w/ renegs >> {versions, ['tlsv1.2', 'tlsv1.1']}, % add tlsv1 if you must >> {reuse_sessions, false}, % drop session cache for perf >> {ecc, EllipticCurves} % see below >> ]. >> > > Forgot to add {honor_ecc_order, true} to that list if you use the ecc option! From codewiget95@REDACTED Wed Aug 30 20:03:46 2017 From: codewiget95@REDACTED (code wiget) Date: Wed, 30 Aug 2017 14:03:46 -0400 Subject: [erlang-questions] How safe is it to leave an open SSL port on the public internet? In-Reply-To: <20170829222354.GB51868@ferdmbp.local> References: <55FFE7D1-782C-4FEF-B23D-BE292BCA5574@gmail.com> <1DAAF3BE-D958-45C8-AA75-2F71EC5D9E68@gmail.com> <20170829221959.GA51868@ferdmbp.local> <20170829222354.GB51868@ferdmbp.local> Message-ID: <18BC0FBC-2A70-4D1B-A3F8-5E022B4DF45C@gmail.com> Also, Fred, I re-read your post and wanted to either start a quick discussion/warn you about elliptic curves. According to the NSA: "the growth of elliptic curve use has bumped up against the fact of continued progress in the research on quantum computing, which has made it clear that elliptic curve cryptography is not the long term solution many once hoped it would be.? The NSA has deprecated ECC, whether or not that means that some foreign actor has a crack or if they are that worried about quantum computing is to be seen, but for now it seems like we should be moving away from ECC. -------------- next part -------------- An HTML attachment was scrubbed... URL: From derekbrown121@REDACTED Wed Aug 30 22:30:34 2017 From: derekbrown121@REDACTED (Derek Brown) Date: Wed, 30 Aug 2017 13:30:34 -0700 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons Message-ID: Looking for thoughts on the various Erlang HTTP client libraries, for a use case involving, say, 3000-5000 calls per second (from my server, as a client, to HTTP servers). I'm thinking of httpc, Gun, Hackney, any others... Performance, reliability, other pros/cons. Involves both GET and POST requests, with a hard roundtrip timeout of ~100 ms. Thanks for any thoughts. Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From puzza007@REDACTED Wed Aug 30 23:09:33 2017 From: puzza007@REDACTED (Paul Oliver) Date: Wed, 30 Aug 2017 21:09:33 +0000 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: References: Message-ID: Hi Derek, You could also take a look at https://github.com/puzza007/katipo (disclaimer - I'm the author). It uses libcurl/libevent, so is highly compatible and very fast (coming second in https://github.com/lpgauth/httpc_bench. Cheers, Paul. On Thu, Aug 31, 2017 at 8:44 AM Derek Brown wrote: > Looking for thoughts on the various Erlang HTTP client libraries, for a > use case involving, say, 3000-5000 calls per second (from my server, as a > client, to HTTP servers). I'm thinking of httpc, Gun, Hackney, any > others... Performance, reliability, other pros/cons. Involves both GET and > POST requests, with a hard roundtrip timeout of ~100 ms. Thanks for any > thoughts. > > Derek > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin@REDACTED Wed Aug 30 23:47:44 2017 From: martin@REDACTED (Martin Karlsson) Date: Thu, 31 Aug 2017 09:47:44 +1200 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: References: Message-ID: <20170830214744.jbwrfs2tbncgqdje@littledevil.rpk> I have done some benchmarking for our specific use case. * Main web server with high number of concurrent requests coming in (~1000 concurrent requests per second but benchmarked up to 10,000 requests) * Most requests requires a call out to an back-end HTTP server (only GET) * There are few back-end endpoints (3). It is not clear how many different kind of servers you are going to call out to and this will likely affect your choice of library. For our use case I benchmarked httpc, gun, hackney, dlhttpc, ibrowse. All of them behaves well when the endpoint replies quickly. However, once the latency of the response goes up we have problems with all "traditional clients". The only one that worked for us in this use case was dlhttpc (https://github.com/ferd/dlhttpc) and this is likely because of the dispcount "router". I know hackney has experimental dispcount pooling so that would likely work as well. * httpc is known to have a few quirks. For our use case which only do HTTP get requests, no cookies, pipelines or other it works well but there is a gen_server bottleneck in there somewhere when calling only few endpoints (have not tried with many endpoints). The biggest pro is that it is built-in, but behaves a bit strange under load. * gun - was discarded quickly as I couldn't get anywhere near the performance I was after and I didn't have the time to trouble-shoot with so many other alternatives. * hackney and ibrowse - seemed like the most stable clients in term of response times. hackney is also used lots in the elixir community so it should have some good production usage. ibrowse has been around for a long time. * dlhttpc - works best for us. Not updated in a while so can't say anything for more advanced usage. Again, this is becuase of dispcount routing, not the actual http client in it self. Cheers Martin From ok@REDACTED Thu Aug 31 00:34:16 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 31 Aug 2017 10:34:16 +1200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170830064202.GA17946@erix.ericsson.se> References: <20170829083528.GA91917@erix.ericsson.se> <20170830064202.GA17946@erix.ericsson.se> Message-ID: <5aead032-c932-48db-3453-329e043780aa@cs.otago.ac.nz> On 30/08/17 6:42 PM, Raimo Niskanen wrote: > On Wed, Aug 30, 2017 at 11:44:57AM +1200, Richard A. O'Keefe wrote: >> There are applications of random numbers for which it is important >> that 0 never be returned. Of course, nothing stops me writing > > What kind of applications? I would like to get a grip on how needed this > function is? I'll give you just one example. There are two ways I know to generate normally distributed random numbers. One of them goes like this: sqrt(-2 * ln(randnz()) * cos(pi * random()) where random() is in [0,1) but randnz() must be in (0,1). OK, I'll give you another example. This is part of an algorithm for generating gamma variates, one of the best known. U <- random() if U <= r then z <- -ln(U/r) else z <- ln(random()/lambda) end You will notice that both of the calls to ln will go wrong if random() can return 0. These aren't the only examples, but I have an appointment. From frank.muller.erl@REDACTED Thu Aug 31 03:34:34 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 31 Aug 2017 01:34:34 +0000 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: <20170830214744.jbwrfs2tbncgqdje@littledevil.rpk> References: <20170830214744.jbwrfs2tbncgqdje@littledevil.rpk> Message-ID: The above mentioned clients are also slow compared to "buoy": https://github.com/lpgauth/buoy Watch the author's video at: http://www.erlang-factory.com/sfbay2017/louisphilippe-gauthier.html /Frank Le mer. 30 ao?t 2017 ? 23:47, Martin Karlsson wrote: > I have done some benchmarking for our specific use case. > > * Main web server with high number of concurrent requests coming in > (~1000 concurrent requests per second but benchmarked up to 10,000 > requests) > * Most requests requires a call out to an back-end HTTP server (only > GET) > * There are few back-end endpoints (3). > > It is not clear how many different kind of servers you are going to call > out to and this will likely affect your choice of library. > > For our use case I benchmarked httpc, gun, hackney, dlhttpc, ibrowse. > > All of them behaves well when the endpoint replies quickly. However, > once the latency of the response goes up we have problems with all > "traditional > clients". The only one that worked for us in this use case was dlhttpc > (https://github.com/ferd/dlhttpc) and this is likely because of the > dispcount "router". > > I know hackney has experimental dispcount pooling so that would likely > work as well. > > * httpc is known to have a few quirks. For our use case which only do HTTP > get requests, no cookies, pipelines or other it works well but there is > a gen_server bottleneck in there somewhere when calling only few > endpoints (have not tried with many endpoints). The biggest pro is that > it is built-in, but behaves a bit strange under load. > > * gun - was discarded quickly as I couldn't get anywhere near the > performance I was after and I didn't have the time to trouble-shoot > with so many other alternatives. > > * hackney and ibrowse - seemed like the most stable clients in term of > response times. hackney is also used lots in the elixir community so > it should have some good production usage. ibrowse has been around > for a long time. > > * dlhttpc - works best for us. Not updated in a while so can't say > anything for more advanced usage. Again, this is becuase of dispcount > routing, not the actual http client in it self. > > > Cheers > Martin > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Aug 31 06:46:32 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 31 Aug 2017 00:46:32 -0400 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: References: Message-ID: hackney may also fit your needs. it tries to handle much of the specs. and http2/wbsockets are coming, release is planned on 25/09 since it had to be postponed due to some others priorities. how slow it is, isrelative. most of the time your backend on the same host will accept less connections than you except anyway. Hence the need of pools. And most of the time the speed is more a matter of your application internals . hackney is used in prod since a long time on large applications. There are some speedup coming in hackney when you disable some options. no figures yet. otherwise patches are also welcome. If you have some questions feel free to contact me. Benoit On 31 August 2017 at 03:34:34, Frank Muller (frank.muller.erl@REDACTED) wrote: > The above mentioned clients are also slow compared to "buoy": > > https://github.com/lpgauth/buoy > > Watch the author's video at: > > http://www.erlang-factory.com/sfbay2017/louisphilippe-gauthier.html > > /Frank > > Le mer. 30 ao?t 2017 ? 23:47, Martin Karlsson > wrote: > >> I have done some benchmarking for our specific use case. >> >> * Main web server with high number of concurrent requests coming in >> (~1000 concurrent requests per second but benchmarked up to 10,000 >> requests) >> * Most requests requires a call out to an back-end HTTP server (only >> GET) >> * There are few back-end endpoints (3). >> >> It is not clear how many different kind of servers you are going to call >> out to and this will likely affect your choice of library. >> >> For our use case I benchmarked httpc, gun, hackney, dlhttpc, ibrowse. >> >> All of them behaves well when the endpoint replies quickly. However, >> once the latency of the response goes up we have problems with all >> "traditional >> clients". The only one that worked for us in this use case was dlhttpc >> (https://github.com/ferd/dlhttpc) and this is likely because of the >> dispcount "router". >> >> I know hackney has experimental dispcount pooling so that would likely >> work as well. >> >> * httpc is known to have a few quirks. For our use case which only do HTTP >> get requests, no cookies, pipelines or other it works well but there is >> a gen_server bottleneck in there somewhere when calling only few >> endpoints (have not tried with many endpoints). The biggest pro is that >> it is built-in, but behaves a bit strange under load. >> >> * gun - was discarded quickly as I couldn't get anywhere near the >> performance I was after and I didn't have the time to trouble-shoot >> with so many other alternatives. >> >> * hackney and ibrowse - seemed like the most stable clients in term of >> response times. hackney is also used lots in the elixir community so >> it should have some good production usage. ibrowse has been around >> for a long time. >> >> * dlhttpc - works best for us. Not updated in a while so can't say >> anything for more advanced usage. Again, this is becuase of dispcount >> routing, not the actual http client in it self. >> >> >> Cheers >> Martin >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Aug 31 07:01:30 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 31 Aug 2017 08:01:30 +0300 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: References: Message-ID: I've looked into your katipo and I do not understand what for is it. You launch curl as a separate program and communicate with it from erlang. What for? Erlang has excellent network communication system and you have selected most inefficient way to fetch http. We in Flussonic use lhttpc and it works for us. Some fixes like adding timing and internal protocol support and it works for us. -------------- next part -------------- An HTML attachment was scrubbed... URL: From puzza007@REDACTED Thu Aug 31 08:04:11 2017 From: puzza007@REDACTED (Paul Oliver) Date: Thu, 31 Aug 2017 06:04:11 +0000 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: References: Message-ID: On Thu, Aug 31, 2017 at 5:01 PM Max Lapshin wrote: > I've looked into your katipo and I do not understand what for is it. > It's a way to use libcurl from Erlang. You can find more information about libcurl here https://curl.haxx.se/libcurl/. > > You launch curl as a separate program and communicate with it from erlang. > What for? > HTTP compatibility (curl) and safety (port program). > > Erlang has excellent network communication system and you have selected > most inefficient way to fetch http. > As mentioned in my earlier email, the httpc_benchmark ranks katipo fairly well so any inefficiency of using an external port doesn't seem to be taking too much of a toll. I've considered using some sort of NIF but haven't found the need. Pull requests welcome though. > > > > We in Flussonic use lhttpc and it works for us. Some fixes like adding > timing and internal protocol support and it works for us. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From puzza007@REDACTED Thu Aug 31 09:22:19 2017 From: puzza007@REDACTED (Paul Oliver) Date: Thu, 31 Aug 2017 07:22:19 +0000 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: References: Message-ID: On Thu, Aug 31, 2017 at 6:20 PM Taras Halturin wrote: > I think, Max means that you choose most expensive way to deal with it and > it's not about efficiency of http handling but about efficiency of aim > achieving :) > > The most expensive way in terms of what? If not speed do you mean development effort? Given that the aim is to use libcurl then the choice is a port executable or some sort of NIF. When using a port executable I don't have to worry about it crashing my VM and all I pay is the price of port communications. If I use a NIF I have to concern myself with making sure my NIF code and the code in libcurl doesn't crash my VM. That's a lot more development time and risk. Cheers, Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From halturin@REDACTED Thu Aug 31 08:20:10 2017 From: halturin@REDACTED (Taras Halturin) Date: Thu, 31 Aug 2017 09:20:10 +0300 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: References: Message-ID: I think, Max means that you choose most expensive way to deal with it and it's not about efficiency of http handling but about efficiency of aim achieving :) On Thu, Aug 31, 2017 at 9:04 AM, Paul Oliver wrote: > > > On Thu, Aug 31, 2017 at 5:01 PM Max Lapshin wrote: > >> I've looked into your katipo and I do not understand what for is it. >> > > It's a way to use libcurl from Erlang. You can find more information about > libcurl here https://curl.haxx.se/libcurl/. > > >> >> You launch curl as a separate program and communicate with it from >> erlang. What for? >> > > HTTP compatibility (curl) and safety (port program). > > >> >> Erlang has excellent network communication system and you have selected >> most inefficient way to fetch http. >> > > As mentioned in my earlier email, the httpc_benchmark ranks katipo fairly > well so any inefficiency of using an external port doesn't seem to be > taking too much of a toll. I've considered using some sort of NIF but > haven't found the need. Pull requests welcome though. > > >> >> >> >> We in Flussonic use lhttpc and it works for us. Some fixes like adding >> timing and internal protocol support and it works for us. >> > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Best Regards. Taras Halturin -------------- next part -------------- An HTML attachment was scrubbed... URL: From palani@REDACTED Thu Aug 31 09:26:07 2017 From: palani@REDACTED (Palanikumar Gopalakrishnan) Date: Thu, 31 Aug 2017 12:56:07 +0530 Subject: [erlang-questions] Enable verbose in erlang Message-ID: I need to enable verbose in erlang. To find out the working of erlang module and what are the submodules are called. -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Aug 31 09:30:34 2017 From: zxq9@REDACTED (zxq9) Date: Thu, 31 Aug 2017 16:30:34 +0900 Subject: [erlang-questions] Enable verbose in erlang In-Reply-To: References: Message-ID: <2890118.qB6HYZNPLq@changa> On 2017?08?31? ??? 12:56:07 Palanikumar Gopalakrishnan wrote: > I need to enable verbose in erlang. To find out the working of erlang > module and what are the submodules are called. It sounds like you need "tracing", not "verbose". There are many forms of this. Can you describe your original problem? -Craig From karlsson.rm@REDACTED Thu Aug 31 10:07:01 2017 From: karlsson.rm@REDACTED (Mikael Karlsson) Date: Thu, 31 Aug 2017 10:07:01 +0200 Subject: [erlang-questions] +secio emulator flag Message-ID: Hi, I noticed that the secio emulator flag (scheduler eager check for io) default value has changed from false to true since ERTS 7.0. Could someone explain what this means in practice, I do not quite understand the line: "This, however, also implies that execution of outstanding I/O operation is not prioritized to the same extent as when false is passed." Is there any special cases where you would prefer to still have it set to false. /Mikael -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Aug 31 11:41:42 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 31 Aug 2017 11:41:42 +0200 Subject: [erlang-questions] Erlang HTTP client libraries- pros/cons In-Reply-To: References: Message-ID: <02a31d28-4b89-e64c-d8a8-3c6925c91f51@ninenines.eu> On 08/30/2017 10:30 PM, Derek Brown wrote: > Looking for thoughts on the various Erlang HTTP client libraries, for a > use case involving, say, 3000-5000 calls per second (from my server, as > a client, to HTTP servers). I'm thinking of httpc, Gun, Hackney, any > others... Performance, reliability, other pros/cons. Involves both GET > and POST requests, with a hard roundtrip timeout of ~100 ms. Thanks for > any thoughts. Gun is not a general purpose client, it is made exclusively for long running connections (for example, connect when the node starts, and stay connected forever). It requires a lot more setup efforts than others because of this (pooling, circuit breakers and other are for you to do). Sameroom.io is an example service using it. Regardless of clients, I would recommend using HTTP/2 as much as possible if performance is a concern and you're doing more than one request per connection. The gains are really significant. -- Lo?c Hoguin https://ninenines.eu From erlang@REDACTED Thu Aug 31 13:11:42 2017 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 31 Aug 2017 13:11:42 +0200 Subject: [erlang-questions] Building a DAW in Erlang Message-ID: Hello, I want to build a "proof of concept" DAW (Digital Audio Workstation) Why? - just for fun. DAWs involve complex GUIs, complex audio processing, and complex man-machine interactions - I'd like to make the DAW from small well defined isolated communicating components. I was wondering about audio - has anybody ideas about this. I want to build a synthesizer with as much Erlang as possible and as little C as necessary - and even if I use C I'd like reference algorithms in Erlang to validate the C. Has anybody implemented any digital audio filters in Erlang? The Synthesizer would have no GUI (The GUI is a separate issue). Having got to the point where I can generate raw PCM samples I need to output these - preferably send them to a socket, with an audio server that reads and outputs the samples - has anybody done this? Initially I'll do this on mac but would eventually like it to run on windows and linux. All idea are very welcome /Joe From raimo+erlang-questions@REDACTED Thu Aug 31 14:30:08 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 31 Aug 2017 14:30:08 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <5aead032-c932-48db-3453-329e043780aa@cs.otago.ac.nz> References: <20170829083528.GA91917@erix.ericsson.se> <20170830064202.GA17946@erix.ericsson.se> <5aead032-c932-48db-3453-329e043780aa@cs.otago.ac.nz> Message-ID: <20170831123008.GA73645@erix.ericsson.se> On Thu, Aug 31, 2017 at 10:34:16AM +1200, Richard A. O'Keefe wrote: > > > On 30/08/17 6:42 PM, Raimo Niskanen wrote: > > On Wed, Aug 30, 2017 at 11:44:57AM +1200, Richard A. O'Keefe wrote: > > >> There are applications of random numbers for which it is important > >> that 0 never be returned. Of course, nothing stops me writing > > > > What kind of applications? I would like to get a grip on how needed this > > function is? > > I'll give you just one example. There are two ways I know to generate > normally distributed random numbers. One of them goes like this: > > sqrt(-2 * ln(randnz()) * cos(pi * random()) > > where random() is in [0,1) but randnz() must be in (0,1). > > OK, I'll give you another example. This is part of an algorithm for > generating gamma variates, one of the best known. > > U <- random() > if U <= r then > z <- -ln(U/r) > else > z <- ln(random()/lambda) > end > > You will notice that both of the calls to ln will go wrong if > random() can return 0. > > These aren't the only examples, but I have an appointment. Thank you! Should I make a pull request of this? https://github.com/erlang/otp/compare/OTP-20.0...RaimoNiskanen:raimo/stdlib/rand-uniformNZ Is the name uniformNZ good enough? Are uniform floats complete enough with this addition? -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From palani@REDACTED Thu Aug 31 14:04:10 2017 From: palani@REDACTED (Palanikumar Gopalakrishnan) Date: Thu, 31 Aug 2017 17:34:10 +0530 Subject: [erlang-questions] How to convert sql return data to json Message-ID: I want to get data from mysql and convert them into json format. I tried with this module https://sourceforge.net/projects/erlmysql/files/latest/download?source=directory Previously i tried this modules, https://github.com/Eonblast/Emysql This module give support for json conversion. But It was not support for Transaction. So I need to convert Sql return data to Json format. Please guide me to solve this challenge -- *Warm Regards,* *Palanikumar Gopalakrishnan *[image: ?] *Developer* -------------- next part -------------- An HTML attachment was scrubbed... URL: From nayibor@REDACTED Thu Aug 31 15:12:48 2017 From: nayibor@REDACTED (Nuku Ameyibor) Date: Thu, 31 Aug 2017 13:12:48 +0000 Subject: [erlang-questions] How to convert sql return data to json In-Reply-To: References: Message-ID: <59a80b52.0259810a.f6398.c6ee@mx.google.com> You can use https://github.com/talentdeficit/jsx, https://github.com/davisp/jiffy to convert erlang terms to and from json . -----Original Message----- From: "Palanikumar Gopalakrishnan" Sent: ?8/?31/?2017 12:32 PM To: "erlang-questions@REDACTED" Subject: [erlang-questions] How to convert sql return data to json I want to get data from mysql and convert them into json format. I tried with this module https://sourceforge.net/projects/erlmysql/files/latest/download?source=directory Previously i tried this modules, https://github.com/Eonblast/Emysql This module give support for json conversion. But It was not support for Transaction. So I need to convert Sql return data to Json format. Please guide me to solve this challenge -- Warm Regards, Palanikumar Gopalakrishnan Developer -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Aug 31 15:13:46 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 31 Aug 2017 13:13:46 +0000 Subject: [erlang-questions] How to convert sql return data to json In-Reply-To: References: Message-ID: The way Erlang systems handle JSON data is the following: 1. In Erlang, every value passed around is an "Erlang Term". 2. A subset of these terms are representable in JSON. 3. JSON encoders such as 'jiffy' or 'jsx' are able to take the valid subset from 2 and convert it into a binary containing the equivalent JSON representation. Your problem can be solved by taking the data you obtain from MySQL and then convert the data into a form which follows the valid JSON subset and has the structure you want. Then call something like `jsx:encode(Data)` on your `Data`. Some background on 2 above: Erlang terms can contain tuples such as {ok, 37} or {error, not_connected}. These do not have a representation in JSON (since it lacks the equivalent of a symbol/atom and tuples). So most JSON encoders reject Erlang terms containing tuples as subterms. The details depend on the encoder though. I can't remember if MySQL is able to return a result set directly as JSON, and if it were, you have to check that the driver you use can handle such a result as well--I can't help you much in this case, unfortunately. Hopefully this will get you started on a solution! On Thu, Aug 31, 2017 at 2:32 PM Palanikumar Gopalakrishnan < palani@REDACTED> wrote: > I want to get data from mysql and convert them into json format. I tried > with this > > module > https://sourceforge.net/projects/erlmysql/files/latest/download?source=directory > > > Previously i tried this modules, > https://github.com/Eonblast/Emysql > > This module give support for json conversion. But It was not support for > Transaction. > > So I need to convert Sql return data to Json format. Please guide me to > solve this challenge > > > > > > > -- > > *Warm Regards,* > > *Palanikumar Gopalakrishnan *[image: ?] > *Developer* > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Aug 31 15:42:06 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 31 Aug 2017 15:42:06 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <13223505.cVJpAap13P@changa> References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> Message-ID: <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> On 08/30/2017 12:33 PM, zxq9 wrote: > On 2017?08?30? ??? 10:29:12 Raimo Niskanen wrote: > >> It is in general safer to change the documentation to match the reality. > > Wow. I certainly hope this is not the general policy for OTP. We program against the documentation. The documentation *is* our reality. It also seems it's not even listed in the release notes. We program against the documentation, if the documentation has a breaking changes it would be great to know about it. -- Lo?c Hoguin https://ninenines.eu From jesper.louis.andersen@REDACTED Thu Aug 31 16:20:55 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 31 Aug 2017 14:20:55 +0000 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> Message-ID: On Thu, Aug 31, 2017 at 3:42 PM Lo?c Hoguin wrote: > > I certainly hope this is not the general policy for OTP. We program > against the documentation. The documentation *is* our reality. > > I think it is fair to evaluate on a case by case basis. Some times, the documentation and the implementation are not matching up. This means either the documentation or the implementation is wrong (not xor here!). Which of which is wrong depends a bit on the case, and there are definitely borderline situations where it is very hard to determine which way you should let the thing fall. I don't think you can make blanket statements on which way you should lean because there are good counterexamples in both "camps" so to speak. Another view is that the documentation is the specification. But again, both the specification and the implementation can be wrong and some times the correct operation is to change the specification. When I worked with formal semantics, it was quite common that you altered the specification in ways that let you prove a meta-theoretic property about the specification. Not altering it would simply make the proof way to complicated and hard. Perhaps even impossible. It is an extreme variant of letting the documentation match the reality in a certain sense. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Aug 31 16:56:58 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 31 Aug 2017 16:56:58 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> Message-ID: On 08/31/2017 04:20 PM, Jesper Louis Andersen wrote: > On Thu, Aug 31, 2017 at 3:42 PM Lo?c Hoguin > wrote: > > > I certainly hope this is not the general policy for OTP. We program > against the documentation. The documentation *is* our reality. > > > I think it is fair to evaluate on a case by case basis. I'm not arguing against that, but rather against that it's "in general safer to change the documentation". It should be a case by case basis but it's also important to recognize that users write software against the documentation, and to take this into account when making breaking changes. To give an example, if such a thing were to happen in Cowboy, which follows semver, two cases could happen: * The documentation is wrong but the next version is a patch release: fix the code to match the documentation. The rule is: don't break people's programs. * The documentation is wrong but the next version is a major release: fix the documentation AND announce it as a breaking change (with all details; and probably release a patch release for the old version as above). The rule is: breaking people's programs is OK, just make sure you tell them about it! > I don't think you can make blanket statements on which way you should > lean because there are good counterexamples in both "camps" so to speak. Properly matching people's expectations is a lot more important than whatever counterexamples may exist. -- Lo?c Hoguin https://ninenines.eu From attila.r.nohl@REDACTED Thu Aug 31 17:13:34 2017 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Thu, 31 Aug 2017 17:13:34 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> Message-ID: 2017-08-31 15:42 GMT+02:00 Lo?c Hoguin : > On 08/30/2017 12:33 PM, zxq9 wrote: >> >> On 2017?08?30? ??? 10:29:12 Raimo Niskanen wrote: >> >>> It is in general safer to change the documentation to match the reality. >> >> >> Wow. > > > I certainly hope this is not the general policy for OTP. We program against > the documentation. The documentation *is* our reality. I disagree. Take this example: https://lwn.net/SubscriberLink/732420/9b9f8f2825f1877f/ The printk() function in the Linux kernel was documented to print new logs to new lines unless the KERN_CONT option was passed. In reality it didn't always started new lines and people expected (maybe even relied on) this - and when the code was updated to match the documentation, they were genuinely surprised when their code was broken. From zxq9@REDACTED Thu Aug 31 17:19:26 2017 From: zxq9@REDACTED (zxq9) Date: Fri, 01 Sep 2017 00:19:26 +0900 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> Message-ID: <2360714.W4GS8jAmS5@changa> On 2017?08?31? ??? 14:20:55 Jesper Louis Andersen wrote: > On Thu, Aug 31, 2017 at 3:42 PM Lo?c Hoguin wrote: > > > > > I certainly hope this is not the general policy for OTP. We program > > against the documentation. The documentation *is* our reality. > > > > > I think it is fair to evaluate on a case by case basis. Some times, the > documentation and the implementation are not matching up. This means either > the documentation or the implementation is wrong (not xor here!). Which of > which is wrong depends a bit on the case, and there are definitely > borderline situations where it is very hard to determine which way you > should let the thing fall. > > I don't think you can make blanket statements on which way you should lean > because there are good counterexamples in both "camps" so to speak. > > Another view is that the documentation is the specification. But again, > both the specification and the implementation can be wrong and some times > the correct operation is to change the specification. When I worked with > formal semantics, it was quite common that you altered the specification in > ways that let you prove a meta-theoretic property about the specification. > Not altering it would simply make the proof way to complicated and hard. > Perhaps even impossible. It is an extreme variant of letting the > documentation match the reality in a certain sense. The other route is to make existing functions do what they say they are going to whenever possible, add functions that provide the prescribed functionality, and deprecate and annotate (with warnings where appropriate) the ones that cannot provide whatever they originally claimed to. And be quite noisy about all of this. OTP has many, many examples of this. It prevents surprise breakage of old code that depends on some particular (and occasionally peculiar) behavior while forging a path ahead -- allowing users to make an informed decision to review and update old code or stick with an older version of the runtime (which tends to be the more costly choice in many cases, but at least it can be an informed decision). Consider what happened with now/0, for example. Now we have a more complex family of time functions but never was it viewed as an acceptable approach to simply shift documentation around a bit here and there in a rather quiet manner while adding in contextual execution features (that is to say, hidden states) that would cause now/0 to behave in a new way. And now/0 is deprecated. > I think it is fair to evaluate on a case by case basis. OK. I'll buy that. In an EXTREMELY limited number of cases you will have a function that simply cannot live up to its spec without a ridiculous amount of nitpicky work that wouldn't really matter to anyone. This is not one of those cases. And in this case we are talking about providing a largely pure API in the standard library, not some meta behavior that acts indirectly through a rewrite system based on some proofing mechanics where the effects of improper definitions are magnified with each transformation. So I get what you're saying, but this is not one of those cases, and for those odd cases it is much safer to deprecate functions, mark them as unsafe, provide compiler warnings and so on if the situation is just THAT BAD, and write a new function that is properly documented in a way that won't suddenly change later. For a functional language's standard library the majority of functions aren't going to be magically tricky, and specs are concrete promises while implementations ephemeral. At least this change happened in a major release, not a minor one. If it is forgivable anywhere, it is in a major release. The tricky bit is that the promises a language's standard libs make to authors are a bit more sticky than those made by separate libraries provided in a given language. And yes, that is at least as much part of the social contract inherent in the human part of the programming world as it is a part of the technical contract implicit in published documentation. The social part of the contract is more important, from what I've seen. Consider why Ruby and many previously popular JS frameworks are considered to be cancer now -- its not just because things changed, it is that the way they changed jerked people around. The issue I am addressing is a LOT more important than whether `0 =< X < 1.0`, of course (yeah, on this one issue, we'll figure it out). It is a general attitude that is absolutely dangerous. >> It is in general safer to change the documentation to match the reality. This is as corrosive a statement as can be. We need to think very carefully about that before this sort of thinking starts becoming common in other areas of OTP in general. -Craig From essen@REDACTED Thu Aug 31 17:27:02 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 31 Aug 2017 17:27:02 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> Message-ID: On 08/31/2017 05:13 PM, Attila Rajmund Nohl wrote: > 2017-08-31 15:42 GMT+02:00 Lo?c Hoguin : >> On 08/30/2017 12:33 PM, zxq9 wrote: >>> >>> On 2017?08?30? ??? 10:29:12 Raimo Niskanen wrote: >>> >>>> It is in general safer to change the documentation to match the reality. >>> >>> >>> Wow. >> >> >> I certainly hope this is not the general policy for OTP. We program against >> the documentation. The documentation *is* our reality. > > I disagree. Take this example: > https://lwn.net/SubscriberLink/732420/9b9f8f2825f1877f/ The printk() > function in the Linux kernel was documented to print new logs to new > lines unless the KERN_CONT option was passed. In reality it didn't > always started new lines and people expected (maybe even relied on) > this - and when the code was updated to match the documentation, they > were genuinely surprised when their code was broken. This story is not about people following the documentation and then have the documentation be "fixed" under their feet without them noticing, it is in fact the complete opposite. -- Lo?c Hoguin https://ninenines.eu From zxq9@REDACTED Thu Aug 31 17:27:24 2017 From: zxq9@REDACTED (zxq9) Date: Fri, 01 Sep 2017 00:27:24 +0900 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <2360714.W4GS8jAmS5@changa> References: <2360714.W4GS8jAmS5@changa> Message-ID: <3169768.bf5bcCL3Qv@changa> On 2017?09?01? ??? 00:19:26 zxq9 wrote: > `0 =< X < 1.0` DHOH! Yes, I know, I know... 0.0 =< X < 1.0 /(>.<)\ From attila.r.nohl@REDACTED Thu Aug 31 17:32:44 2017 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Thu, 31 Aug 2017 17:32:44 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> Message-ID: 2017-08-31 17:27 GMT+02:00 Lo?c Hoguin : > On 08/31/2017 05:13 PM, Attila Rajmund Nohl wrote: >> >> 2017-08-31 15:42 GMT+02:00 Lo?c Hoguin : [...] >>> I certainly hope this is not the general policy for OTP. We program >>> against >>> the documentation. The documentation *is* our reality. >> >> >> I disagree. Take this example: >> https://lwn.net/SubscriberLink/732420/9b9f8f2825f1877f/ The printk() >> function in the Linux kernel was documented to print new logs to new >> lines unless the KERN_CONT option was passed. In reality it didn't >> always started new lines and people expected (maybe even relied on) >> this - and when the code was updated to match the documentation, they >> were genuinely surprised when their code was broken. > > > This story is not about people following the documentation and then have the > documentation be "fixed" under their feet without them noticing, it is in > fact the complete opposite. The moral of the story: people are programming against behavior/implementation, not documentation. In these cases fixing the implementation instead of the documentation has very real possibility of breaking existing programs. Of course, one can tell its users that "it's your fault you haven't followed the documentation!" but it doesn't necessarily make those users happy... From essen@REDACTED Thu Aug 31 17:35:45 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 31 Aug 2017 17:35:45 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> Message-ID: <8a3ec5fb-27a6-78dc-aa60-e9650c1445b4@ninenines.eu> On 08/31/2017 05:32 PM, Attila Rajmund Nohl wrote: > 2017-08-31 17:27 GMT+02:00 Lo?c Hoguin : >> On 08/31/2017 05:13 PM, Attila Rajmund Nohl wrote: >>> >>> 2017-08-31 15:42 GMT+02:00 Lo?c Hoguin : > [...] >>>> I certainly hope this is not the general policy for OTP. We program >>>> against >>>> the documentation. The documentation *is* our reality. >>> >>> >>> I disagree. Take this example: >>> https://lwn.net/SubscriberLink/732420/9b9f8f2825f1877f/ The printk() >>> function in the Linux kernel was documented to print new logs to new >>> lines unless the KERN_CONT option was passed. In reality it didn't >>> always started new lines and people expected (maybe even relied on) >>> this - and when the code was updated to match the documentation, they >>> were genuinely surprised when their code was broken. >> >> >> This story is not about people following the documentation and then have the >> documentation be "fixed" under their feet without them noticing, it is in >> fact the complete opposite. > > The moral of the story: people are programming against > behavior/implementation, not documentation. In these cases fixing the > implementation instead of the documentation has very real possibility > of breaking existing programs. Of course, one can tell its users that > "it's your fault you haven't followed the documentation!" but it > doesn't necessarily make those users happy... Maybe in the Linux kernel. Outside, where there is such a thing as documentation (comments are not documentation), if the code behaves differently than the documentation, you open a ticket... And in that case, yes, for a limited time, you will program against the behavior and not against the documentation. But it's the exception, not the rule. -- Lo?c Hoguin https://ninenines.eu From zxq9@REDACTED Thu Aug 31 17:45:28 2017 From: zxq9@REDACTED (zxq9) Date: Fri, 01 Sep 2017 00:45:28 +0900 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: Message-ID: <2569930.rQmXfUV9Kl@changa> On 2017?08?31? ??? 17:32:44 Attila Rajmund Nohl wrote: > 2017-08-31 17:27 GMT+02:00 Lo?c Hoguin : > > On 08/31/2017 05:13 PM, Attila Rajmund Nohl wrote: > >> > >> 2017-08-31 15:42 GMT+02:00 Lo?c Hoguin : > [...] > >>> I certainly hope this is not the general policy for OTP. We program > >>> against > >>> the documentation. The documentation *is* our reality. > >> > >> > >> I disagree. Take this example: > >> https://lwn.net/SubscriberLink/732420/9b9f8f2825f1877f/ The printk() > >> function in the Linux kernel was documented to print new logs to new > >> lines unless the KERN_CONT option was passed. In reality it didn't > >> always started new lines and people expected (maybe even relied on) > >> this - and when the code was updated to match the documentation, they > >> were genuinely surprised when their code was broken. > > > > > > This story is not about people following the documentation and then have the > > documentation be "fixed" under their feet without them noticing, it is in > > fact the complete opposite. > > The moral of the story: people are programming against > behavior/implementation, not documentation. In these cases fixing the > implementation instead of the documentation has very real possibility > of breaking existing programs. Of course, one can tell its users that > "it's your fault you haven't followed the documentation!" but it > doesn't necessarily make those users happy... There was once a boy who always rode his bike on the right side of the streets in his neighborhood. Sure, the signs all said "keep left" but, well, everyone just ignores the signs where he lives. One day a new sign was in its place that said "keep right". Now what should he do? -Craig From zxq9@REDACTED Thu Aug 31 17:38:44 2017 From: zxq9@REDACTED (zxq9) Date: Fri, 01 Sep 2017 00:38:44 +0900 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: References: Message-ID: <3163117.AtdE8zJdFY@changa> On 2017?08?31? ??? 17:32:44 you wrote: > The moral of the story: people are programming against > behavior/implementation, not documentation. In these cases fixing the > implementation instead of the documentation has very real possibility > of breaking existing programs. Of course, one can tell its users that > "it's your fault you haven't followed the documentation!" but it > doesn't necessarily make those users happy... There was once a boy who always rode his bike on the right side of the streets in his neighborhood. Sure, the signs all said "keep left" but, well, everyone just ignores the signs where he lives. One day a new sign was in its place that said "keep right". Now what should he do? -Craig From l.rieder@REDACTED Thu Aug 31 18:32:52 2017 From: l.rieder@REDACTED (Lukas Rieder) Date: Thu, 31 Aug 2017 18:32:52 +0200 Subject: [erlang-questions] Question about buffer size given to erl_receive_msg in erl_connect Message-ID: Hello everyone, I have a question about the bufsize argument in the erl_receive_msg function provided in erl_connect. I do not quite understand what impact a given bufsize has on the operation of a C-Node. Is it limiting the amount of data that can be transferred to a C-Node? Quickly trying some things out, with a buffer size of 1000 chars, I could send messages to the C-Node back and forth, that are way bigger than a 1000 char buffer. So I suspect the buffer is only a "buffer" and not limiting anything per se. But what is a good bufsize? And when does one use erl_xreceive_msg over erl_receive_msg? I could not find any good examples on the internet on erl_xreceive_msg. The implementations I found that are using erl_receive_msg all are using a bufsize of 1000. I would be very happy, for my own understanding, if someone can give me some hints or explanation on making these choices. Kind regards, Lukas Rieder -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Thu Aug 31 20:35:24 2017 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 31 Aug 2017 11:35:24 -0700 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170831123008.GA73645@erix.ericsson.se> References: <20170829083528.GA91917@erix.ericsson.se> <20170830064202.GA17946@erix.ericsson.se> <5aead032-c932-48db-3453-329e043780aa@cs.otago.ac.nz> <20170831123008.GA73645@erix.ericsson.se> Message-ID: <59A856EC.1030000@gmail.com> On 08/31/2017 05:30 AM, Raimo Niskanen wrote: > On Thu, Aug 31, 2017 at 10:34:16AM +1200, Richard A. O'Keefe wrote: >> >> On 30/08/17 6:42 PM, Raimo Niskanen wrote: >>> On Wed, Aug 30, 2017 at 11:44:57AM +1200, Richard A. O'Keefe wrote: >>>> There are applications of random numbers for which it is important >>>> that 0 never be returned. Of course, nothing stops me writing >>> What kind of applications? I would like to get a grip on how needed this >>> function is? >> I'll give you just one example. There are two ways I know to generate >> normally distributed random numbers. One of them goes like this: >> >> sqrt(-2 * ln(randnz()) * cos(pi * random()) >> >> where random() is in [0,1) but randnz() must be in (0,1). >> >> OK, I'll give you another example. This is part of an algorithm for >> generating gamma variates, one of the best known. >> >> U <- random() >> if U <= r then >> z <- -ln(U/r) >> else >> z <- ln(random()/lambda) >> end >> >> You will notice that both of the calls to ln will go wrong if >> random() can return 0. >> >> These aren't the only examples, but I have an appointment. > Thank you! > > Should I make a pull request of this? > > https://github.com/erlang/otp/compare/OTP-20.0...RaimoNiskanen:raimo/stdlib/rand-uniformNZ > > Is the name uniformNZ good enough? > Are uniform floats complete enough with this addition? As I argued in the original pull request for these recent 20.0 random number changes, a uniform distribution is much more intuitive if it is inclusive: [0,1] For example, if you are dealing with probabilities, it is simpler to think in percentages from 0.00 to 1.00 An example from the python documentation is at https://docs.python.org/3/library/random.html#random.uniform though they have ambiguity about the highest value due to a rounding problem they have. I have had my own dependency for uniform as [0,1] at https://github.com/okeuday/quickrand/blob/fc5e21ec70ee94dd4ce1c5ee02b55ceea03f9008/src/quickrand.erl#L294-L309 so I have been working around this absence. Though I would assume other people would benefit from the addition of a [0,1] function in Erlang/OTP. Best Regards, Michael From jacob01@REDACTED Thu Aug 31 21:06:52 2017 From: jacob01@REDACTED (Jacob) Date: Thu, 31 Aug 2017 21:06:52 +0200 Subject: [erlang-questions] Building a DAW in Erlang In-Reply-To: References: Message-ID: <6c9f874e-50dd-9d7f-c106-d07b07c9bcff@gmx.net> Hello Joe, On 31.08.2017 13:11, Joe Armstrong wrote: > Hello, > > I want to build a "proof of concept" DAW (Digital Audio Workstation) > > Why? - just for fun. > > DAWs involve complex GUIs, complex audio processing, and complex > man-machine interactions - I'd like to make the DAW from small well > defined isolated communicating components. > > I was wondering about audio - has anybody ideas about this. A big challenge with a (distributed) setting consisting of isolated components is to get synchronisation right, especially with an asynchronous architecture. If you think of hardware components connected via e.g. AES/EBU unidirectional digital links, each of these components usually gets a clock source (either audio or wordclock input) unless it is the clock master. When doing audio with a computer, the clock master is in general the audio sound card. If you want to have a distributed system with _several_ sound cards (even on the same host), you will need a hardware wordclock sync between them. Otherwise you would introduce new clock domains. If the system clock is used to generate audio samples (lets say to generate 48000 samples per second) and the samples are just fed to the sound card, evil things will happen (which things exactly depends mainly on the hardware and on drivers, but they will be audible in general which I consider evil). The cause is: the system clock is in another clock domain than the audio card. And system oscillators are in general cheap ones with relatively bad stability and temperature coefficients. When different clock domains are involved, sample-rate conversion and DLL/PLL and stuff like that are needed, with in turn may introduce audible pitch changes (DVB for instance relies on a reference clock of 27MHz with a max change (1st derivation) of 75mHz/s to avoid this, see ISO/IEC 13818-1). Nevertheless adaptive sample rate conversion is done e.g. by vlc and other players to play broadcast radio/television e.g. via RTP. I'd rather avoid doing that myself (I did it once and its no fun). So when thinking about a system open for professional use, I'd design it to use a single clock domain. Two approaches to achieve this on a computer based system are 1. broadcasting or distributing a reference clock (e.g. with a sample index) 2. using backpressure I think the former is a good fit for an asynchronous (Erlang) system. On the other hand, one could start with an RTP sender (e.g. payload type 10 or 11, see https://en.wikipedia.org/wiki/RTP_audio_video_profile ) and rely on the system clock as a quick start (both encapsulated in a single process, accepting timestamped audio and distributing timestamped clock references). Perhaps vlc can be convinced to send RTP/RTCP based on the audio input of the same audio interface, that would be a better time reference (same clock domain). The more pro audio open source variant would probably be to use jackd, e.g. by writing an Erlang netjack1 client (https://github.com/jackaudio/jackaudio.github.com/wiki/WalkThrough_User_NetJack). > > All idea are very welcome That's jump a dump of what came to my mind about your idea. I hope it's useful. Jacob From jesper.louis.andersen@REDACTED Thu Aug 31 21:08:30 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 31 Aug 2017 19:08:30 +0000 Subject: [erlang-questions] Question about buffer size given to erl_receive_msg in erl_connect In-Reply-To: References: Message-ID: Hi, In C, a buffer is represented by a pointer. Pointers do not carry size information, and there is no means by which you know where the end-of-buffer is. Hence, it is C-idiomatic to supply another argument, bufsize, indicating the size of the buffer the pointer is pointing at. If you give a too small bufsize, you are not going to use up the whole buffer. If you give a too large bufsize, you will overwrite data after the buffer in memory. This is wrong and usually leads to program crashes. Also, allocation checkers will detect this for you and warn you that your program is wrong. If the message does not fit into the buffer, the call will fail and set and error indicating that the message is too large to fit inside the buffer. You could then try to extend the buffer and try again. The xreceive call assumes the buffer is allocated by malloc. If it doesn't fit, the call will create a buffer large enough for the message and reallocate. This is why the types are different: you supply the addresses of the buffer pointer and the bufsize counter. Hence the call can update the contents of those addresses with the newly formed buffer, should it decide to move them. The reason both calls exist is because it isn't always the case that the buffer will be allocated by malloc on the heap. You may statically allocate the buffer, or allocate it on the stack, for instance. On Thu, Aug 31, 2017 at 8:18 PM Lukas Rieder wrote: > Hello everyone, > > I have a question about the bufsize argument in the erl_receive_msg > function provided in erl_connect. > > I do not quite understand what impact a given bufsize has on the operation > of a C-Node. > > Is it limiting the amount of data that can be transferred to a C-Node? > > Quickly trying some things out, with a buffer size of 1000 chars, I could > send messages to the C-Node back and forth, that are way bigger than a 1000 > char buffer. > So I suspect the buffer is only a "buffer" and not limiting anything per > se. > > But what is a good bufsize? > And when does one use erl_xreceive_msg over erl_receive_msg? > > I could not find any good examples on the internet on erl_xreceive_msg. > The implementations I found that are using erl_receive_msg all are using a > bufsize of 1000. > I would be very happy, for my own understanding, if someone can give me > some hints or explanation on making these choices. > > Kind regards, > > > Lukas Rieder > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Aug 31 22:30:08 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 31 Aug 2017 16:30:08 -0400 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <8a3ec5fb-27a6-78dc-aa60-e9650c1445b4@ninenines.eu> References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> <8a3ec5fb-27a6-78dc-aa60-e9650c1445b4@ninenines.eu> Message-ID: <20170831203007.GE51868@ferdmbp.local> On 08/31, Lo?c Hoguin wrote: >Maybe in the Linux kernel. Outside, where there is such a thing as >documentation (comments are not documentation), if the code behaves >differently than the documentation, you open a ticket... And in that >case, yes, for a limited time, you will program against the behavior >and not against the documentation. But it's the exception, not the >rule. > I think 'it depends' truly *is* the best way to go about it. Let's see a few examples: - A function does what is in the doc, but also does a bit more at the same time. Do you fix by removing the additional functionality people may now rely on, or by updating the doc to match the implementation? - The documentation specifies that by sending the atom 'tsl1.3' you can set up a TLS 1.3 connection, but the implementation only accepts 'tsl1.3' and crashes on 'tsl1.3'. Do you not update the documentation for what was a typo, or do you add support for 'tsl1.3' as a parameter? If anybody relied on that behaviour, they relied on the code not working! - A function for socket handling returns values such as `{error, emfile | enfile | econnrefused}`. Someone finds out that the syscalls it relays data to also may return `{error, eperm | eaccess}` on top of what was specified before. Do you swallow the errors and mask them, or update the docs? Or is it not just a bug in the docs? - A function's doc says it sorts in a stable manner but it does not. Which one should you change? There seems to be no winner on this one. - A function logs information while it operates on data, but the documentation makes no reference to it. Someone working with it in a specific environment has issues with that behaviour.* There's plenty of cases where the doc and the implementation may be wrong individually. Either as a mistake on either sides, by omission, or through a bug. Usually you have to take a pragmatic approach wondering which of the fixes will give the lowest conflict or impact to people using the code. Now is the case of the random behaviour similar to specifying a bad type boundary similar to "not supporting all the errors", to a breach of a well-established contract, etc.? That's a really hard question, but saying either "the doc always wins" or "the code always wins" unequivocally sounds like a real bad policy to me. Regards, Fred. * In adding shell history to Erlang, we got tripped up on disk_log doing just that. We added a new option to force it to be silent when needed, for example, so both the code and the doc required a fix! From essen@REDACTED Thu Aug 31 23:57:04 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 31 Aug 2017 23:57:04 +0200 Subject: [erlang-questions] On OTP rand module difference between OTP 19 and OTP 20 In-Reply-To: <20170831203007.GE51868@ferdmbp.local> References: <2691824.Matf7LgGbk@changa> <20170830082912.GA29138@erix.ericsson.se> <13223505.cVJpAap13P@changa> <77c901fb-88b1-f2e6-23c0-a2eeaaa690e4@ninenines.eu> <8a3ec5fb-27a6-78dc-aa60-e9650c1445b4@ninenines.eu> <20170831203007.GE51868@ferdmbp.local> Message-ID: <60e6bdd0-2833-3896-9c03-382c1c2b9091@ninenines.eu> On 08/31/2017 10:30 PM, Fred Hebert wrote: > On 08/31, Lo?c Hoguin wrote: >> Maybe in the Linux kernel. Outside, where there is such a thing as >> documentation (comments are not documentation), if the code behaves >> differently than the documentation, you open a ticket... And in that >> case, yes, for a limited time, you will program against the behavior >> and not against the documentation. But it's the exception, not the rule. >> > > I think 'it depends' truly *is* the best way to go about it. Well it's a good thing I agreed with this in an email sent almost 6 hours ago then. :-) -- Lo?c Hoguin https://ninenines.eu