From publicityifl@REDACTED Tue May 1 07:26:47 2018 From: publicityifl@REDACTED (Jurriaan Hage) Date: Mon, 30 Apr 2018 22:26:47 -0700 Subject: [erlang-questions] 3rd CfP: IFL 2018 (30th Symposium on Implementation and Application of Functional Languages) Message-ID: Hello, Please, find below the third call for papers for IFL 2018. Please forward these to anyone you think may be interested. Apologies for any duplicates you may receive. best regards, Jurriaan Hage Publicity Chair of IFL --- ============================================================ ==================== IFL 2018 30th Symposium on Implementation and Application of Functional Languages University of Massachusetts Lowell, MA, USA September 5th-7th, 2018 http://iflconference.org ============================================================ ==================== ### Scope The goal of the IFL symposia is to bring together researchers actively engaged in the implementation and application of functional and function-based programming languages. IFL 2018 will be a venue for researchers to present and discuss new ideas and concepts, work in progress, and publication-ripe results related to the implementation and application of functional languages and function-based programming. Topics of interest to IFL include, but are not limited to: - language concepts - type systems, type checking, type inferencing - compilation techniques - staged compilation - run-time function specialization - run-time code generation - partial evaluation - (abstract) interpretation - metaprogramming - generic programming - automatic program generation - array processing - concurrent/parallel programming - concurrent/parallel program execution - embedded systems - web applications - (embedded) domain specific languages - security - novel memory management techniques - run-time profiling performance measurements - debugging and tracing - virtual/abstract machine architectures - validation, verification of functional programs - tools and programming techniques - (industrial) applications ### Keynote Speakers * Adam Chlipala, Massachusetts Institute of Technology CSAIL * Arjun Guha, University of Massachusetts Amherst ### Submissions and peer-review Differently from previous editions of IFL, IFL 2018 solicits two kinds of submissions: * Regular papers (12 pages including references) * Draft papers for presentations ('weak' limit between 8 and 15 pages) Regular papers will undergo a rigorous review by the program committee, and will be evaluated according to their correctness, novelty, originality, relevance, significance, and clarity. A set of regular papers will be conditionally accepted for publication. Authors of conditionally accepted papers will be provided with committee reviews along with a set of mandatory revisions. Regular papers not accepted for publication will be considered as draft papers, at the request of the author. Draft papers will be screened to make sure that they are within the scope of IFL, and will be accepted for presentation or rejected accordingly. Prior to the symposium: Authors of conditionally accepted papers and accepted presentations will submit a pre-proceedings version of their work that will appear in the draft proceedings distributed at the symposium. The draft proceedings does not constitute a formal publication. We require that at least one of the authors present the work at IFL 2018. After the symposium: Authors of conditionally accepted papers will submit a revised versions of their paper for the formal post-proceedings. The program committee will assess whether the mandatory revisions have been adequately addressed by the authors and thereby determines the final accept/reject status of the paper. Our interest is to ultimately accept all conditionally accepted papers. If you are an author of a conditionally accepted paper, please make sure that you address all the concerns of the reviewers. Authors of accepted presentations will be given the opportunity to incorporate the feedback from discussions at the symposium and will be invited to submit a revised full article for the formal post-proceedings. The program committee will evaluate these submissions according to their correctness, novelty, originality, relevance, significance, and clarity, and will thereby determine whether the paper is accepted or rejected. ### Publication The formal proceedings will appear in the International Conference Proceedings Series of the ACM Digital Library. At no time may work submitted to IFL be simultaneously submitted to other venues; submissions must adhere to ACM SIGPLAN's republication policy: http://www.sigplan.org/Resources/Policies/Republication ### Important dates Submission of regular papers: May 25, 2018 Submission of draft papers: July 17, 2018 Regular and draft papers notification: July 20, 2018 Deadline for early registration: August 8, 2018 Submission of pre-proceedings version: August 29, 2018 IFL Symposium: September 5-7, 2018 Submission of papers for post-proceedings: November 7, 2018 Notification of acceptance: December 22, 2018 Camera-ready version: February 10, 2019 ### Submission details All contributions must be written in English. Papers must use the ACM two columns conference format, which can be found at: http://www.acm.org/publications/proceedings-template Authors submit through EasyChair: https://easychair.org/conferences/?conf=ifl2018 ### Peter Landin Prize The Peter Landin Prize is awarded to the best paper presented at the symposium every year. The honored article is selected by the program committee based on the submissions received for the formal review process. The prize carries a cash award equivalent to 150 Euros. ### Organization and Program committee Chairs: Jay McCarthy & Matteo Cimini, University of Massachusetts Lowell, USA Program Committee: * Arthur Chargueraud, Inria, FR * Ben Delaware, Purdue University, USA * Christos Dimoulas, Northwestern University, USA * David Darais, University of Vermont, USA * Dominic Orchard, University of Kent, UK * Ekaterina Komendantskaya, Heriot-Watt University, UK * Garrett Morris, University of Kansas, USA * Heather Miller, EPFL & Northeastern University, CH & USA * Jeremy Yallop, University of Cambridge, UK * Keiko Nakata, SAP Innovation Center Potsdam, DE * Laura Castro, University of A Coruna, ESP * Magnus Myreen, Chalmers University of Technology, SWE * Natalia Chechina, Bournemouth University, UK * Peter Achten, Radboud Universiteit Nijmegen, NL * Peter-Michael Osera, Grinnell College, USA * Richard Eisenberg, Bryn Mawr College, USA * Trevor McDonell, University of New South Wales, AUS * Yukiyoshi Kameyama, University of Tsukuba, JAP ### Venue The 30th IFL is organized by the University of Massachusetts Lowell. The City of Lowell is located at the heart of the Merrimack Valley just 30 miles northwest of Boston. Lowell can be easily reached by train or taxi. See the website for more information on the venue. ### Acknowledgments This call-for-papers is an adaptation and evolution of content from previous instances of IFL. We are grateful to prior organizers for their work, which is reused here. A part of IFL 2018 format and CFP language that describes conditionally accepted papers has been adapted from call-for-papers of OOPSLA conferences. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Tue May 1 15:05:25 2018 From: gerhard@REDACTED (Gerhard Lazu) Date: Tue, 1 May 2018 14:05:25 +0100 Subject: [erlang-questions] How can we increase multiblock carrier utilization for binary_alloc? Message-ID: We are running on Erlang 20.3.4 on Linux 4.4.0-119-generic 14.04.1-Ubuntu SMP x86_64. These are the flags that we use for beam.smp: /var/vcap/packages/erlang-20.3.4/lib/erlang/erts-9.3/bin/beam.smp -W w -A 64 -zdbbl 128000 -K true -stbt db -zdbbl 128000 -P 1048576 -t 5000000 -MHas ageffcbf -MBas ageffcbf -MHlmbcs 512 -MBlmbcs 512 When the node boots, the binary_alloc multiblock carrier utilization is ~99% (90MB allocated, 90MB used). [1] As the load on the node starts, binary_alloc mbcs util drops to ~78% (~800MB allocated, ~625MB used). [2] When load stops completely, binary_alloc mbcs util drops to ~61% (~55MB allocated, ~35MB used) & mbcs_pool goes to ~36% (~215MB allocated, ~80MB used). In the context of RabbitMQ, this is a big problem since memory usage controls whether incoming messages are blocked or not (a.k.a. memory alarm). It's essential that the Erlang VM utilises memory as efficiently as possible, otherwise nodes under no load can remain blocked permanently. [3] Our goal is for the Erlang VM to have as little unused memory as possible. As you could see in the referenced screenshots [1][2][3], the total unused memory starts at ~30MB and grows to ~280MB. Considering that the total RSS memory that the beam.smp process uses is 545MB [3], half of it goes unused (~280MB), and this is a big problem for RabbitMQ, as mentioned above. Lukas, you've shared some excellent documentation in the past around the topic of memory management in Erlang. I am wondering if you have deeper/more refined insights that could help our current challenge. In Erlang Memory Management Battle Stories [5], on slide 29, you mention "Decreasing largest mbc size will make more carriers and hopefully be able to free them". In our case, it doesn't, and I'm hoping that you can point us in the right direction. Ferd, I've studied your past battles with Erlang's memory management, and I can only thank you for sharing so much over the years. Erlang in Anger [6] and recon [7] helped immensely, thank you. I would appreciate greatly if you could nudge us in the right direction, maybe we've missed something. During this exploration, there's a specific thing that's been bugging us: why is RSS smaller than the allocated memory? Thank you all, Gerhard & Lo?c. [1]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/mbcs_utilization_after_boot.png [2]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/mbcs_utilisation_under_load.png [3]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/mbcs_utilisation_after_load.png [4]: http://erlang.org/doc/man/erts_alloc.html [5]: http://www.erlang-factory.com/static/upload/media/139454517145429lukaslarsson.pdf [6]: https://www.erlang-in-anger.com/ [7]: https://github.com/ferd/recon -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Tue May 1 15:16:54 2018 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Tue, 1 May 2018 15:16:54 +0200 Subject: [erlang-questions] [ANN] jason 1.0.2 In-Reply-To: <1525022708.2635058.1354711824.162071BB@webmail.messagingengine.com> References: <139f0ee3-2170-5b8c-2409-1e5c659a2522@wanadoo.fr> <1525022708.2635058.1354711824.162071BB@webmail.messagingengine.com> Message-ID: <2ee3a4bb-ae78-9876-c717-e79557cbfbe2@wanadoo.fr> Le 29/04/2018 ? 19:25, Tristan Sloughter a ?crit?: > >> Note: this project has nothing to do with Elixir project with same name. > > I was going to reply to bring that up. Especially since there is a hex package for jason, the Elixir app, it will be confusing. Even without a name change you should publish to hex though :), then at least people searching for jason will see both and pick the right one. You can publish with a different package name than the app name, so like jason_erl. Hi Tristan, thanks for your comment. I'm not an Elixir guy and lately see that another project had same same on hex. Yes I will consider publish on hex with a slightly different name, but waiting for a feedback on my project first, and maybe add some fixes if needed before. Honestly I do not like the syntax of mix.exs, always a pain for me, and prefers Cargo.toml for Rust or composer.json for Composer PHP syntaxes ... More generally speaking, a decentralized package manager based on block-chain and maybe IPFS would be a better alternative for futur. If I had more time ... Regards From t@REDACTED Tue May 1 16:54:38 2018 From: t@REDACTED (Tristan Sloughter) Date: Tue, 01 May 2018 07:54:38 -0700 Subject: [erlang-questions] [ANN] jason 1.0.2 In-Reply-To: <2ee3a4bb-ae78-9876-c717-e79557cbfbe2@wanadoo.fr> References: <139f0ee3-2170-5b8c-2409-1e5c659a2522@wanadoo.fr> <1525022708.2635058.1354711824.162071BB@webmail.messagingengine.com> <2ee3a4bb-ae78-9876-c717-e79557cbfbe2@wanadoo.fr> Message-ID: <1525186478.2513256.1356967320.5FDCDEA0@webmail.messagingengine.com> You can publish with rebar3, no need to have a mix.exs. https://hex.pm/docs/rebar3_publish https://www.rebar3.org/v3.0/docs/hex-package-management Assuming that is why you brought up mix.exs. And just to throw it out there, at the very beginning of rebar3 I actually toyed with toml (was using cargo and lein as inspiration) for the config. It didn't seem worth it. A language that doesn't have lists and tuples as basic structures it makes sense, felt like unnecessary indirection for Erlang. -- Tristan Sloughter "I am not a crackpot" - Abe Simpson t@REDACTED On Tue, May 1, 2018, at 6:16 AM, PAILLEAU Eric wrote: > Le 29/04/2018 ? 19:25, Tristan Sloughter a ?crit?: > > > >> Note: this project has nothing to do with Elixir project with same name. > > > > I was going to reply to bring that up. Especially since there is a hex package for jason, the Elixir app, it will be confusing. Even without a name change you should publish to hex though :), then at least people searching for jason will see both and pick the right one. You can publish with a different package name than the app name, so like jason_erl. > > Hi Tristan, > thanks for your comment. > I'm not an Elixir guy and lately see that another project had same same > on hex. > Yes I will consider publish on hex with a slightly different name, but > waiting for a feedback on my project first, and maybe add some fixes if > needed before. > > Honestly I do not like the syntax of mix.exs, always a pain for me, > and prefers Cargo.toml for Rust or composer.json for Composer PHP > syntaxes ... > > More generally speaking, a decentralized package manager based on > block-chain and maybe IPFS would be a better alternative for futur. > If I had more time ... > > Regards > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zzantozz@REDACTED Tue May 1 19:23:52 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Tue, 01 May 2018 17:23:52 +0000 Subject: [erlang-questions] TLS handshake records don't handle fragments? Message-ID: I've been getting handshake_failure alerts when trying to connect to a particular server, and I think I've traced it to the fact that the TLS records aren't being handled correctly with respect to fragments. In particular, this server is sending a rather large "certificate request" to allow for client cert auth, and the list is too long to fit in one TLS record. That's breaking the TLS handshake in at least Erlang 18 and 19, I believe. It's basically a mirror image of the problem described in https://bugs.erlang.org/browse/ERL-83. That issue is with Erlang as the TLS server. I'm seeing the same thing with it being the client. Is this addressed somewhere? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Wed May 2 12:32:28 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 2 May 2018 12:32:28 +0200 Subject: [erlang-questions] TLS handshake records don't handle fragments? In-Reply-To: References: Message-ID: Hi! 2018-05-01 19:23 GMT+02:00 Ryan Stewart : > I've been getting handshake_failure alerts when trying to connect to a > particular server, and I think I've traced it to the fact that the TLS > records aren't being handled correctly with respect to fragments. In > particular, this server is sending a rather large "certificate request" to > allow for client cert auth, and the list is too long to fit in one TLS > record. That's breaking the TLS handshake in at least Erlang 18 and 19, I > believe. It's basically a mirror image of the problem described in > https://bugs.erlang.org/browse/ERL-83. That issue is with Erlang as the > TLS server. I'm seeing the same thing with it being the client. Is this > addressed somewhere? > > Can you give me a possibility to recreate the issue? That issue you described was fixed in 18 and both the client and the server uses the same code to encode handshakes. The issue was in the encoding and not in the decoding. Can you tell us more details of how you reached your conclusion? Regards Ingela Erlang/OTP Team - Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed May 2 13:21:40 2018 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 2 May 2018 13:21:40 +0200 Subject: [erlang-questions] How can we increase multiblock carrier utilization for binary_alloc? In-Reply-To: References: Message-ID: Hello, On Tue, May 1, 2018 at 3:05 PM, Gerhard Lazu wrote: > We are running on Erlang 20.3.4 on Linux 4.4.0-119-generic 14.04.1-Ubuntu > SMP x86_64. These are the flags that we use for beam.smp: > > /var/vcap/packages/erlang-20.3.4/lib/erlang/erts-9.3/bin/beam.smp -W w -A > 64 -zdbbl 128000 -K true -stbt db -zdbbl 128000 -P 1048576 -t 5000000 -MHas > ageffcbf -MBas ageffcbf -MHlmbcs 512 -MBlmbcs 512 > > When the node boots, the binary_alloc multiblock carrier utilization is > ~99% (90MB allocated, 90MB used). [1] > > As the load on the node starts, binary_alloc mbcs util drops to ~78% > (~800MB allocated, ~625MB used). [2] > > When load stops completely, binary_alloc mbcs util drops to ~61% (~55MB > allocated, ~35MB used) & mbcs_pool goes to ~36% (~215MB allocated, ~80MB > used). > > In the context of RabbitMQ, this is a big problem since memory usage > controls whether incoming messages are blocked or not (a.k.a. memory > alarm). It's essential that the Erlang VM utilises memory as efficiently as > possible, otherwise nodes under no load can remain blocked permanently. [3] > > Our goal is for the Erlang VM to have as little unused memory as possible. > As you could see in the referenced screenshots [1][2][3], the total unused > memory starts at ~30MB and grows to ~280MB. Considering that the total RSS > memory that the beam.smp process uses is 545MB [3], half of it goes unused > (~280MB), and this is a big problem for RabbitMQ, as mentioned above. > Why do you not use erlang:memory() as the base for whether you can accept more messages? Having a low memory utilisation is not bad in itself, unless of course some other program on the same machine needs the memory. Looking at the used memory under load and after, at peak the allocated memory is 1510 MB and then after it is 577 MB. So about 2/3rds of the allocated memory was returned to the OS. While this is not perfect, it is not terrible either. Reducing it further may not be easy. > > Lukas, you've shared some excellent documentation in the past around the > topic of memory management in Erlang. I am wondering if you have > deeper/more refined insights that could help our current challenge. In > Erlang Memory Management Battle Stories [5], on slide 29, you mention > "Decreasing largest mbc size will make more carriers and hopefully be able > to free them". In our case, it doesn't, and I'm hoping that you can point > us in the right direction. > Can you reproduce the behaviour? Would it be possible to get a recon_alloc snaphot during and after load? Have you seen https://github.com/erlang/otp/pull/1790 that was just merged to master with the accompanying blog post: http://blog.erlang.org/Memory-instrumentation-in-OTP-21/? I assume that you have tried ageffcaobf? > > During this exploration, there's a specific thing that's been bugging us: > why is RSS smaller than the allocated memory? > Every time I try to understand how RSS works I just end up getting more confused. > > Thank you all, Gerhard & Lo?c. > > [1]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-all > ocators/mbcs_utilization_after_boot.png > [2]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-all > ocators/mbcs_utilisation_under_load.png > [3]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-all > ocators/mbcs_utilisation_after_load.png > [4]: http://erlang.org/doc/man/erts_alloc.html > [5]: http://www.erlang-factory.com/static/upload/media/1394545171 > 45429lukaslarsson.pdf > [6]: https://www.erlang-in-anger.com/ > [7]: https://github.com/ferd/recon > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik.x.nord@REDACTED Wed May 2 13:37:59 2018 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Wed, 2 May 2018 11:37:59 +0000 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) Message-ID: <1525261079.4069.73.camel@ericsson.com> OTP 21 Release Candidate 1 This is the first of two planned release candidates before the OTP 21 release. The intention whit this release is that you as users try it and give us feedback if something does not work as expected. Could be a bug,an unexpected incompatibility, a significant change of characteristics in a negative direction, etc.? Erlang/OTP 21 is a new major release with new features, improvements as well as incompatibilities. Potential Incompatibilities ?* All Corba applications are now moved from the OTP repository ????* A new Corba repository will be created https://github.com/erlang ?* New applications ftp and tftp, moved from inets ?* ssl no longer supports 3_DES cipher suites or RSA-key exchange cipher suites by default ?* erlang:monitor on a primitive node (erl_interface, jinterface, etc) will no longer fail with badarg exception. Instead a monitor will be created, but it will only supervise the connection to the node. ?Highlights ?Erts: ????* Enhanced IO scalability ????* Support for usage of distribution controller processes for ??????alternative transports, routing etc ????* compact instructions on 64bit systems for code below 4GB 20% less ??????memory for loaded code ????* Rewrite of the efile-driver with NIFs and "Dirty schedulers" ??????resulting in faster file operations ????* non-smp VM removed ????* link and monitor optimized for scalability ????* os:getenv/putenv now work on thread-safe emulation. No longer in ??????sync with libc getenv(3). Manual synchronization will be needed. Compiler: ????* Misc compiler optimizations including contributions from the ??????Elixir team resulting in 10% improvements in benchmarks ????* "Tuple calls" have been removed from the run-time system. ????* Code such as f({ok, Val}) -> {ok, Val} is now automatically ??????rewritten to f({ok, Val} = Tuple) -> Tuple. this reduces code ??????size, execution time, and removed GC pressure. ????* More information in stacktrace from a number of operators ????* erlang:get_stacktrace/0 deprecated to be replaced with try ... ??????catch C:R:Stacktrace -> ... ????* Creation of small maps with literal keys optimized. Security: ????* DTLS support in SSL ????* Enhanced support for distribution over TLS ????* "unsecure" ciphers removed from defaults in SSL and SSH. ????* A new option value defined to facilitate implementing exec ??????servers. Old option kept for compatibility, but now gives errors ??????on stderror. Standard libraries: ????* New API for logging, logger ????* New uri_string module for parsing URIs according to "The ??????standard" ????* New function lists:search(list,fun/1) -> {ok, Value} | false ????* Changed default behaviour of .erlang loading. escript, erlc, ??????dialyzer and typer no longer load an .erlang at all. For more details see http://erlang.org/download/otp_src_21.0-rc1.readme Pre built versions for Windows can be fetched here: http://erlang.org/download/otp_win32_21.0-rc1.exe http://erlang.org/download/otp_win64_21.0-rc1.exe Online documentation can be browsed here: http://erlang.org/documentation/doc-10.0-rc1/doc/ The Erlang/OTP source can also be found at GitHub on the official Erlang repository, https://github.com/erlang/otp with tag OTP-21.0-rc1 Thank you for all your contributions! From jesper.louis.andersen@REDACTED Wed May 2 14:51:09 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 02 May 2018 12:51:09 +0000 Subject: [erlang-questions] How can we increase multiblock carrier utilization for binary_alloc? In-Reply-To: References: Message-ID: On Tue, May 1, 2018 at 3:06 PM Gerhard Lazu wrote: > > During this exploration, there's a specific thing that's been bugging us: > why is RSS smaller than the allocated memory? > > This seems fairly obvious to me, but perhaps I am missing something. The Erlang system has allocated memory from the kernel, but the kernel has not yet handed that memory out to the process, and hence it is not in the RSS (Resident Set Size). As you hit new pages, there should be kernel traps, a page is allocated to the process (bumping RSS) and the program is resumed. If you allocate a larger carrier the "inner parts" of it might not be allocated before the first access. The other situation is that you have excessive memory pressure and the system starts removing pages which are possible to remove (they either bear no data, have already been written to the page/swap file on disk, have libraries in them, or madvise(2) have been called and told the OS that the pages are not needed[0]). In general, most systems with a GC doesn't give memory back to the OS straight away. They either keep it indefinitely (perhaps with an madvise(2) on the areas they don't need), or they have a reaper later on which can give back fully unused "spans" of memory (Go does this, for instance, but it isn't instant). [0] Be *very* cautious with madvise(2) since its implementation semantics are slightly different on Linux/BSD/Illumos. especially around DONTNEED/FREE and friends, which have different semantics. In particular, if a don't need a page right now, is the operating system allowed to hand me a zeroed page later if I start going for that memory again. Bryan Cantrill has some interesting pointers w.r.t. Lx-branded Illumos Zones and this :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fchschneider@REDACTED Wed May 2 15:06:36 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Wed, 2 May 2018 15:06:36 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <1525261079.4069.73.camel@ericsson.com> References: <1525261079.4069.73.camel@ericsson.com> Message-ID: <71efcc86-cb05-d649-1895-b2f905cafcce@gmail.com> Is the fun/1 really the second argument in the call lists:search/2? lists:search(list,fun/1) -> {ok, Value} | false Doesn't feel consistent with the other functions in the list module which have the fun/1 as the first argument. Cheers, Frans On 05/02/2018 01:37 PM, Henrik Nord X wrote: > lists:search(list,fun/1) -> {ok, Value} | false From dm.klionsky@REDACTED Wed May 2 15:10:39 2018 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 2 May 2018 16:10:39 +0300 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <71efcc86-cb05-d649-1895-b2f905cafcce@gmail.com> References: <1525261079.4069.73.camel@ericsson.com> <71efcc86-cb05-d649-1895-b2f905cafcce@gmail.com> Message-ID: <6a04cc3d-58f3-2c81-58ff-c1cf2de297a3@gmail.com> No, it's fine. https://github.com/erlang/otp/blob/master/lib/stdlib/src/lists.erl#L1402-L1413 On 05/02/2018 04:06 PM, Frans Schneider wrote: > Is the fun/1 really the second argument in the call lists:search/2? > > lists:search(list,fun/1) -> {ok, Value} | false > > Doesn't feel consistent with the other functions in the list module > which have the fun/1 as the first argument. > > Cheers, > > Frans > > On 05/02/2018 01:37 PM, Henrik Nord X wrote: >> lists:search(list,fun/1) -> {ok, Value} | false > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- BR, Dmitry From zzantozz@REDACTED Wed May 2 16:38:26 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Wed, 02 May 2018 14:38:26 +0000 Subject: [erlang-questions] TLS handshake records don't handle fragments? In-Reply-To: References: Message-ID: Hi Ingela! Yeah, let me be more clear. The issue is when Erlang is acting as a client and receives a "certificate request" from a server during the handshake, as described at https://tools.ietf.org/html/rfc5246#section-7.4.4. Since Erlang is receiving the message, it's a decode rather than an encode, but the problem is that it's not handling fragmented messages, similar to the story I linked. I'm able to recreate the issue easily by starting a server that sends certificate requests using the system-installed CAs on a CentOS box, like this: $ openssl req -x509 -newkey rsa:2048 -keyout private.pem -out public.pem -days 365 -nodes $ openssl s_server -accept 12345 -cert public.pem -key private.pem -CAfile /etc/pki/tls/certs/ca-bundle.crt -verify 1 Then from Erlang: 1> application:ensure_all_started(ssl). {ok,[crypto,asn1,public_key,ssl]} 2> ssl:connect("localhost", 12345, []). =ERROR REPORT==== 2-May-2018::14:30:13 === SSL: certify: tls_connection.erl:774:Fatal error: handshake failure {error,{tls_alert,"handshake failure"}} There are so many CAs included by default these days, it's enough to trigger this bug. If you add a "-msg" to the s_server, you'll see the (very long) certificate request sent out, followed by a handshake_failure alert received from the client (Erlang). On the off chance you try this on a system without enough CAs, I guess we could just generate a bunch of certs and make our own, big CA file. The certificate request in my test above was 19185 bytes, which is larger than a TLS record is allowed to be, so it forces fragmentation. On Wed, May 2, 2018 at 5:32 AM Ingela Andin wrote: > Hi! > > 2018-05-01 19:23 GMT+02:00 Ryan Stewart : > >> I've been getting handshake_failure alerts when trying to connect to a >> particular server, and I think I've traced it to the fact that the TLS >> records aren't being handled correctly with respect to fragments. In >> particular, this server is sending a rather large "certificate request" to >> allow for client cert auth, and the list is too long to fit in one TLS >> record. That's breaking the TLS handshake in at least Erlang 18 and 19, I >> believe. It's basically a mirror image of the problem described in >> https://bugs.erlang.org/browse/ERL-83. That issue is with Erlang as the >> TLS server. I'm seeing the same thing with it being the client. Is this >> addressed somewhere? >> >> > Can you give me a possibility to recreate the issue? That issue you > described was fixed in 18 and both the client and the server uses the same > code to encode handshakes. The issue was in the encoding and not in the > decoding. Can you tell us more details of how you reached your conclusion? > > Regards Ingela Erlang/OTP Team - Ericsson AB > > > > > >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Wed May 2 17:52:08 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 2 May 2018 17:52:08 +0200 Subject: [erlang-questions] TLS handshake records don't handle fragments? In-Reply-To: References: Message-ID: Hi! I have not had time to check, but it sounds like it could be another bug that I recently fixed. Could you try out the release candidate for OTP 21. Regards Ingela Erlang/OTP Team - Ericsson AB 2018-05-02 16:38 GMT+02:00 Ryan Stewart : > Hi Ingela! Yeah, let me be more clear. The issue is when Erlang is acting > as a client and receives a "certificate request" from a server during the > handshake, as described at https://tools.ietf.org/html/ > rfc5246#section-7.4.4. Since Erlang is receiving the message, it's a > decode rather than an encode, but the problem is that it's not handling > fragmented messages, similar to the story I linked. I'm able to recreate > the issue easily by starting a server that sends certificate requests using > the system-installed CAs on a CentOS box, like this: > > $ openssl req -x509 -newkey rsa:2048 -keyout private.pem -out public.pem > -days 365 -nodes > $ openssl s_server -accept 12345 -cert public.pem -key private.pem -CAfile > /etc/pki/tls/certs/ca-bundle.crt -verify 1 > > Then from Erlang: > > 1> application:ensure_all_started(ssl). > {ok,[crypto,asn1,public_key,ssl]} > 2> ssl:connect("localhost", 12345, []). > > =ERROR REPORT==== 2-May-2018::14:30:13 === > SSL: certify: tls_connection.erl:774:Fatal error: handshake failure > {error,{tls_alert,"handshake failure"}} > > There are so many CAs included by default these days, it's enough to > trigger this bug. If you add a "-msg" to the s_server, you'll see the (very > long) certificate request sent out, followed by a handshake_failure alert > received from the client (Erlang). On the off chance you try this on a > system without enough CAs, I guess we could just generate a bunch of certs > and make our own, big CA file. The certificate request in my test above was > 19185 bytes, which is larger than a TLS record is allowed to be, so it > forces fragmentation. > > On Wed, May 2, 2018 at 5:32 AM Ingela Andin > wrote: > >> Hi! >> >> 2018-05-01 19:23 GMT+02:00 Ryan Stewart : >> >>> I've been getting handshake_failure alerts when trying to connect to a >>> particular server, and I think I've traced it to the fact that the TLS >>> records aren't being handled correctly with respect to fragments. In >>> particular, this server is sending a rather large "certificate request" to >>> allow for client cert auth, and the list is too long to fit in one TLS >>> record. That's breaking the TLS handshake in at least Erlang 18 and 19, I >>> believe. It's basically a mirror image of the problem described in >>> https://bugs.erlang.org/browse/ERL-83. That issue is with Erlang as the >>> TLS server. I'm seeing the same thing with it being the client. Is this >>> addressed somewhere? >>> >>> >> Can you give me a possibility to recreate the issue? That issue you >> described was fixed in 18 and both the client and the server uses the same >> code to encode handshakes. The issue was in the encoding and not in the >> decoding. Can you tell us more details of how you reached your conclusion? >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> >> >> >> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From zzantozz@REDACTED Wed May 2 18:08:10 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Wed, 02 May 2018 16:08:10 +0000 Subject: [erlang-questions] TLS handshake records don't handle fragments? In-Reply-To: References: Message-ID: I actually just checked it in the current release of 20, and it works there. It's just broken in 18 and 19 that I know of. On Wed, May 2, 2018 at 10:52 AM Ingela Andin wrote: > Hi! > > I have not had time to check, but it sounds like it could be another bug > that I recently fixed. Could you try out the release candidate for OTP 21. > > Regards Ingela Erlang/OTP Team - Ericsson AB > > 2018-05-02 16:38 GMT+02:00 Ryan Stewart : > >> Hi Ingela! Yeah, let me be more clear. The issue is when Erlang is acting >> as a client and receives a "certificate request" from a server during the >> handshake, as described at >> https://tools.ietf.org/html/rfc5246#section-7.4.4. Since Erlang is >> receiving the message, it's a decode rather than an encode, but the problem >> is that it's not handling fragmented messages, similar to the story I >> linked. I'm able to recreate the issue easily by starting a server that >> sends certificate requests using the system-installed CAs on a CentOS box, >> like this: >> >> $ openssl req -x509 -newkey rsa:2048 -keyout private.pem -out public.pem >> -days 365 -nodes >> $ openssl s_server -accept 12345 -cert public.pem -key private.pem >> -CAfile /etc/pki/tls/certs/ca-bundle.crt -verify 1 >> >> Then from Erlang: >> >> 1> application:ensure_all_started(ssl). >> {ok,[crypto,asn1,public_key,ssl]} >> 2> ssl:connect("localhost", 12345, []). >> >> =ERROR REPORT==== 2-May-2018::14:30:13 === >> SSL: certify: tls_connection.erl:774:Fatal error: handshake failure >> {error,{tls_alert,"handshake failure"}} >> >> There are so many CAs included by default these days, it's enough to >> trigger this bug. If you add a "-msg" to the s_server, you'll see the (very >> long) certificate request sent out, followed by a handshake_failure alert >> received from the client (Erlang). On the off chance you try this on a >> system without enough CAs, I guess we could just generate a bunch of certs >> and make our own, big CA file. The certificate request in my test above was >> 19185 bytes, which is larger than a TLS record is allowed to be, so it >> forces fragmentation. >> >> On Wed, May 2, 2018 at 5:32 AM Ingela Andin >> wrote: >> >>> Hi! >>> >>> 2018-05-01 19:23 GMT+02:00 Ryan Stewart : >>> >>>> I've been getting handshake_failure alerts when trying to connect to a >>>> particular server, and I think I've traced it to the fact that the TLS >>>> records aren't being handled correctly with respect to fragments. In >>>> particular, this server is sending a rather large "certificate request" to >>>> allow for client cert auth, and the list is too long to fit in one TLS >>>> record. That's breaking the TLS handshake in at least Erlang 18 and 19, I >>>> believe. It's basically a mirror image of the problem described in >>>> https://bugs.erlang.org/browse/ERL-83. That issue is with Erlang as >>>> the TLS server. I'm seeing the same thing with it being the client. Is this >>>> addressed somewhere? >>>> >>>> >>> Can you give me a possibility to recreate the issue? That issue you >>> described was fixed in 18 and both the client and the server uses the same >>> code to encode handshakes. The issue was in the encoding and not in the >>> decoding. Can you tell us more details of how you reached your >>> conclusion? >>> >>> Regards Ingela Erlang/OTP Team - Ericsson AB >>> >>> >>> >>> >>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From florian.odronitz@REDACTED Wed May 2 19:08:34 2018 From: florian.odronitz@REDACTED (Florian Odronitz) Date: Wed, 2 May 2018 19:08:34 +0200 Subject: [erlang-questions] Memory leak related to "driver_alloc" Message-ID: Hi, I am trying to debug what looks like a memory leak of about 6GB per week (~10kB/s). The node in question gets data from Kafka and Redis, processes it and puts the result back in Redis (~1MB/s in, 0.9MB/s out). When I log into the node I see erlang:memory() reporting 14,6 GB of memory basically all attributed to "system". Digging deeper recon_alloc:memory(allocated_types) finds that basically all the memory is allocated by "driver_alloc". I tried a few things that did not change anything: - force GC on all processes - restart the main application - stop all applications except the bare minimum I thought maybe it is NIFs leaking memory so I tried to run them in isolation for around 30 CPU minutes each encoding/decoding or compressing/decompressing. They all seem fine. Here is a list of them: snappy: https://github.com/fdmanana/snappy-erlang-nif (c4cd1bb35f3a3a399737d64bd03b479817b90e75) jsonx: https://github.com/odo/jsonx (435fc3e9df5c33bf307fa8c95da7a18b07ea49a7) lz4: https://github.com/szktty/erlang-lz4.git (bdbfbedc89c073cd154de5a0691a198be43c539b) I don't fully understand what driver_alloc contains and what could drive its growth. Any thoughts? Thanks, Florian -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Wed May 2 21:12:22 2018 From: askjuise@REDACTED (Alexander Petrovsky) Date: Wed, 2 May 2018 22:12:22 +0300 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <6a04cc3d-58f3-2c81-58ff-c1cf2de297a3@gmail.com> References: <1525261079.4069.73.camel@ericsson.com> <71efcc86-cb05-d649-1895-b2f905cafcce@gmail.com> <6a04cc3d-58f3-2c81-58ff-c1cf2de297a3@gmail.com> Message-ID: Totally agree with Frans Schneider, the arguments order in different modules in stdlib differs from one to another. And I think will be better reorder args in lists:search. For me much more close approach suggested by Chris Okasaki, short summary from SO ( https://stackoverflow.com/questions/5863128/ordering-of-parameters-to-make-use-of-currying ): For languages that support currying and partial-application easily, there > is one compelling series of arguments, originally from Chris Okasaki: > > - *Put the data structure as the last argument* > > Why? You can then compose operations on the data > nicely. > E.g. insert 1 $ insert 2 $ insert 3 $ s. This also helps for functions on > state > . > Standard libraries such as "containers" follow this convention > > . > Alternate arguments are sometimes given to put the data structure first, > so it can be closed over, yielding functions on a static structure (e.g. > lookup) that are a bit more concise. However, the broad consensus seems to > be that this is less of a win, especially since it pushes you towards > heavily parenthesized code. > > - *Put the most varying argument last* > > For recursive functions, it is common to put the argument that varies the > most (e.g. an accumulator) as the last argument, while the argument that > varies the least (e.g. a function argument) at the start. This composes > well with the data structure last style. So, the lists:search(fun/1, list) -> {ok, Value} | false will be more appropriate then lists:search(list, fun/1) -> {ok, Value} | false > ??, 2 ??? 2018 ?. ? 16:10, Dmitry Klionsky : > No, it's fine. > > https://github.com/erlang/otp/blob/master/lib/stdlib/src/ > lists.erl#L1402-L1413 > > > On 05/02/2018 04:06 PM, Frans Schneider wrote: > > Is the fun/1 really the second argument in the call lists:search/2? > > > > lists:search(list,fun/1) -> {ok, Value} | false > > > > Doesn't feel consistent with the other functions in the list module > > which have the fun/1 as the first argument. > > > > Cheers, > > > > Frans > > > > On 05/02/2018 01:37 PM, Henrik Nord X wrote: > >> lists:search(list,fun/1) -> {ok, Value} | false > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -- > BR, > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed May 2 21:22:57 2018 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 02 May 2018 19:22:57 +0000 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <71efcc86-cb05-d649-1895-b2f905cafcce@gmail.com> <6a04cc3d-58f3-2c81-58ff-c1cf2de297a3@gmail.com> Message-ID: The argument order in the email is a typo, which is what Dmitry tried to point out. http://erlang.org/documentation/doc-10.0-rc1/lib/stdlib-3.5/doc/html/lists.html#search-2 It is lists:search(Pred, List). On Wed, 2 May 2018, 21:12 Alexander Petrovsky, wrote: > Totally agree with Frans Schneider, the arguments order in different > modules in stdlib differs from one to another. And I think will be better > reorder args in lists:search. > > For me much more close approach suggested by Chris Okasaki, short summary > from SO ( > https://stackoverflow.com/questions/5863128/ordering-of-parameters-to-make-use-of-currying > ): > > For languages that support currying and partial-application easily, there >> is one compelling series of arguments, originally from Chris Okasaki: >> >> - *Put the data structure as the last argument* >> >> Why? You can then compose operations on the data >> nicely. >> E.g. insert 1 $ insert 2 $ insert 3 $ s. This also helps for functions >> on state >> . >> Standard libraries such as "containers" follow this convention >> >> . >> Alternate arguments are sometimes given to put the data structure first, >> so it can be closed over, yielding functions on a static structure (e.g. >> lookup) that are a bit more concise. However, the broad consensus seems to >> be that this is less of a win, especially since it pushes you towards >> heavily parenthesized code. >> >> - *Put the most varying argument last* >> >> For recursive functions, it is common to put the argument that varies the >> most (e.g. an accumulator) as the last argument, while the argument that >> varies the least (e.g. a function argument) at the start. This composes >> well with the data structure last style. > > > So, the > > lists:search(fun/1, list) -> {ok, Value} | false > > > will be more appropriate then > > lists:search(list, fun/1) -> {ok, Value} | false > > > >> > ??, 2 ??? 2018 ?. ? 16:10, Dmitry Klionsky : > >> No, it's fine. >> >> >> https://github.com/erlang/otp/blob/master/lib/stdlib/src/lists.erl#L1402-L1413 >> >> >> On 05/02/2018 04:06 PM, Frans Schneider wrote: >> > Is the fun/1 really the second argument in the call lists:search/2? >> > >> > lists:search(list,fun/1) -> {ok, Value} | false >> > >> > Doesn't feel consistent with the other functions in the list module >> > which have the fun/1 as the first argument. >> > >> > Cheers, >> > >> > Frans >> > >> > On 05/02/2018 01:37 PM, Henrik Nord X wrote: >> >> lists:search(list,fun/1) -> {ok, Value} | false >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> -- >> BR, >> Dmitry >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang.org@REDACTED Wed May 2 22:19:30 2018 From: erlang.org@REDACTED (Stanislaw Klekot) Date: Wed, 2 May 2018 22:19:30 +0200 Subject: [erlang-questions] Memory leak related to "driver_alloc" In-Reply-To: References: Message-ID: <20180502201930.GA10768@jarowit.net> On Wed, May 02, 2018 at 07:08:34PM +0200, Florian Odronitz wrote: > Hi, > > I am trying to debug what looks like a memory leak of about 6GB per week (~10kB/s). > The node in question gets data from Kafka and Redis, processes it and puts the result back in Redis (~1MB/s in, 0.9MB/s out). > > When I log into the node I see erlang:memory() reporting 14,6 GB of memory basically all attributed to "system". > Digging deeper recon_alloc:memory(allocated_types) finds that basically all the memory is allocated by "driver_alloc". [...] > I thought maybe it is NIFs leaking memory so I tried to run them in > isolation for around 30 CPU minutes each encoding/decoding or > compressing/decompressing. They all seem fine. If it's really driver_alloc(), then I wouldn't search among the NIFs, but among the port drivers. Do you have any? > I don't fully understand what driver_alloc contains and what could drive its growth. driver_alloc() is, according to its documentation, a wrapper for malloc(3), so it's not its own fault. -- Stanislaw Klekot From ingela.andin@REDACTED Thu May 3 08:03:48 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 3 May 2018 08:03:48 +0200 Subject: [erlang-questions] TLS handshake records don't handle fragments? In-Reply-To: References: Message-ID: Hi Ryan! 2018-05-02 18:08 GMT+02:00 Ryan Stewart : > I actually just checked it in the current release of 20, and it works > there. It's just broken in 18 and 19 that I know of. > Ah, ok the only relevant commit that I can think of that could have fixed it is: 1364c7308e17d43d1a2244e3f2bf11cfec3789ef Regards Ingela Erlang/OTP team > > On Wed, May 2, 2018 at 10:52 AM Ingela Andin > wrote: > >> Hi! >> >> I have not had time to check, but it sounds like it could be another bug >> that I recently fixed. Could you try out the release candidate for OTP 21. >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> 2018-05-02 16:38 GMT+02:00 Ryan Stewart : >> >>> Hi Ingela! Yeah, let me be more clear. The issue is when Erlang is >>> acting as a client and receives a "certificate request" from a server >>> during the handshake, as described at https://tools.ietf.org/html/ >>> rfc5246#section-7.4.4. Since Erlang is receiving the message, it's a >>> decode rather than an encode, but the problem is that it's not handling >>> fragmented messages, similar to the story I linked. I'm able to recreate >>> the issue easily by starting a server that sends certificate requests using >>> the system-installed CAs on a CentOS box, like this: >>> >>> $ openssl req -x509 -newkey rsa:2048 -keyout private.pem -out public.pem >>> -days 365 -nodes >>> $ openssl s_server -accept 12345 -cert public.pem -key private.pem >>> -CAfile /etc/pki/tls/certs/ca-bundle.crt -verify 1 >>> >>> Then from Erlang: >>> >>> 1> application:ensure_all_started(ssl). >>> {ok,[crypto,asn1,public_key,ssl]} >>> 2> ssl:connect("localhost", 12345, []). >>> >>> =ERROR REPORT==== 2-May-2018::14:30:13 === >>> SSL: certify: tls_connection.erl:774:Fatal error: handshake failure >>> {error,{tls_alert,"handshake failure"}} >>> >>> There are so many CAs included by default these days, it's enough to >>> trigger this bug. If you add a "-msg" to the s_server, you'll see the (very >>> long) certificate request sent out, followed by a handshake_failure alert >>> received from the client (Erlang). On the off chance you try this on a >>> system without enough CAs, I guess we could just generate a bunch of certs >>> and make our own, big CA file. The certificate request in my test above was >>> 19185 bytes, which is larger than a TLS record is allowed to be, so it >>> forces fragmentation. >>> >>> On Wed, May 2, 2018 at 5:32 AM Ingela Andin >>> wrote: >>> >>>> Hi! >>>> >>>> 2018-05-01 19:23 GMT+02:00 Ryan Stewart : >>>> >>>>> I've been getting handshake_failure alerts when trying to connect to a >>>>> particular server, and I think I've traced it to the fact that the TLS >>>>> records aren't being handled correctly with respect to fragments. In >>>>> particular, this server is sending a rather large "certificate request" to >>>>> allow for client cert auth, and the list is too long to fit in one TLS >>>>> record. That's breaking the TLS handshake in at least Erlang 18 and 19, I >>>>> believe. It's basically a mirror image of the problem described in >>>>> https://bugs.erlang.org/browse/ERL-83. That issue is with Erlang as >>>>> the TLS server. I'm seeing the same thing with it being the client. Is this >>>>> addressed somewhere? >>>>> >>>>> >>>> Can you give me a possibility to recreate the issue? That issue you >>>> described was fixed in 18 and both the client and the server uses the same >>>> code to encode handshakes. The issue was in the encoding and not in the >>>> decoding. Can you tell us more details of how you reached your >>>> conclusion? >>>> >>>> Regards Ingela Erlang/OTP Team - Ericsson AB >>>> >>>> >>>> >>>> >>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hogberg@REDACTED Thu May 3 08:29:43 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Thu, 3 May 2018 06:29:43 +0000 Subject: [erlang-questions] Patch package OTP 20.3.5 released Message-ID: <1525328982.4017.16.camel@ericsson.com> Patch Package:???????????OTP 20.3.5 Git Tag:?????????????????OTP-20.3.5 Date:????????????????????2018-05-03 Trouble Report Id:???????OTP-15034, OTP-15050 Seq num:???????????????? System:??????????????????OTP Release:?????????????????20 Application:?????????????erts-9.3.1, ssl-8.2.6 Predecessor:?????????????OTP 20.3.4 ?Check out the git tag OTP-20.3.5, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- erts-9.3.1 ------------------------------------------------------ ?--------------------------------------------------------------------- ?The erts-9.3.1 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15034????Application(s): erts ???????????????Fixed a crash in heart:get_cmd/0 when the stored ???????????????command was too long. ?Full runtime dependencies of erts-9.3.1: kernel-5.0, sasl-3.0.1, ?stdlib-3.0 ?--------------------------------------------------------------------- ?--- ssl-8.2.6 ------------------------------------------------------- ?--------------------------------------------------------------------- ?Note! The ssl-8.2.6 application can *not* be applied independently of ???????other applications on an arbitrary OTP 20 installation. ???????On a full OTP 20 installation, also the following runtime ???????dependencies have to be satisfied: ???????-- crypto-4.2 (first satisfied in OTP 20.2) ???????-- public_key-1.5 (first satisfied in OTP 20.1) ?--- Fixed Bugs and Malfunctions --- ? OTP-15050????Application(s): ssl ???????????????Proper handling of clients that choose to send an empty ???????????????answer to a certificate request ?Full runtime dependencies of ssl-8.2.6: crypto-4.2, erts-7.0, ?inets-5.10.7, kernel-3.0, public_key-1.5, stdlib-3.2 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- From john.hogberg@REDACTED Thu May 3 09:38:18 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Thu, 3 May 2018 07:38:18 +0000 Subject: [erlang-questions] Memory leak related to "driver_alloc" In-Reply-To: <20180502201930.GA10768@jarowit.net> References: <20180502201930.GA10768@jarowit.net> Message-ID: <1525333097.4793.22.camel@ericsson.com> On Wed, May 02, 2018 at 07:08:34PM +0200, Florian Odronitz wrote: > When I log into the node I see erlang:memory() reporting 14,6 GB of > memory basically all attributed to "system". > Digging deeper recon_alloc:memory(allocated_types) finds that > basically all the memory is allocated by "driver_alloc". If you can try out OTP 21-rc1, the new memory instrumentation features will help you find the culprit: http://blog.erlang.org/Memory-instrumentation-in-OTP-21/ > I don't fully understand what driver_alloc contains and what could > drive its growth. > Any thoughts? `driver_alloc` is used for general allocations in drivers and NIFs, e.g. `enif_alloc()`, `erl_drv_thread_create()`, or `enif_inspect_iovec()`. Regards, John H?gberg From gerhard@REDACTED Thu May 3 12:18:37 2018 From: gerhard@REDACTED (Gerhard Lazu) Date: Thu, 3 May 2018 11:18:37 +0100 Subject: [erlang-questions] How can we increase multiblock carrier utilization for binary_alloc? In-Reply-To: References: Message-ID: Hi Lukas, Why do you not use erlang:memory() as the base for whether you can accept > more messages? Having a low memory utilisation is not bad in itself, unless > of course some other program on the same machine needs the memory. > We used to use erlang:memory(), but we've learned that it doesn't work well in practice [1]. Linux OOM will take action based on RSS, not Erlang allocated memory. > Looking at the used memory under load and after, at peak the allocated > memory is 1510 MB and then after it is 577 MB. So about 2/3rds of the > allocated memory was returned to the OS. While this is not perfect, it is > not terrible either. Reducing it further may not be easy. > Your observation is true and accurate. It's also true and accurate that out of 577MB allocated, 300MB is used & 277MB is unused, meaning that almost half of the allocated memory is not used. I understand that it may not be easy to reduce the unused memory, but all I'm thinking is that whilst this unused memory might seem small in this particular scenario, what happens when the Erlang VM has 60GB allocated? Would it help if we can show the impact of this behaviour on hosts with larger memory usage? > Can you reproduce the behaviour? Would it be possible to get a recon_alloc > snaphot during and after load? > I'm sharing recon_alloc snapshots during & after load for the following: 1. erts_alloc defaults (lmbcs 5120) [2] 2. -MBlmbcs 512 [3] I've also captured a during & after load screenshots of the 2 configurations running side-by-side (left is erts_alloc defaults (lmbcs 5120), right is -MBlmbcs 512) [4]. While our initial configuration used a few more flags, -MHas ageffcbf -MBas ageffcbf -MHlmbcs 512 -MBlmbcs 512, I've kept things as simple as possible on this run, and only used -MBlmbcs 512 Have you seen https://github.com/erlang/otp/pull/1790 that was just merged > to master with the accompanying blog post: http://blog.erlang.org/M > emory-instrumentation-in-OTP-21/? > I haven't, thank you for sharing. We are waiting on Elixir #6611 before we can test against OTP 21.0-rc1 [5]. > I assume that you have tried ageffcaobf? > Yes, we have tried all allocation strategies. ageffcbf resulted in "spikier" CPU and dirty mem writeback, but also sharper drops in dirty mem writeback. Under load, ageffcbf had 1% lower RSS usage, and 2.5% lower unused memory than ageffcaobf. After load however, ageffcbf had 5% lower RSS usage & 4% lower unused memory. In conclusion, ageffcbf proved the best out of all allocation strategies. Here is a side-by-side comparison of -MBas ageffcaobf -MBlmbcs 512 (left) vs -MBas ageffcbf -MBlmbcs 512 (right) [6], and the relevant recon_alloc snapshots [7]. Thank you Lukas for helping out with this, Gerhard. [1] https://github.com/rabbitmq/rabbitmq-server/issues/1223 & https://github.com/rabbitmq/rabbitmq-server/pull/1259#issuecomment-308428057 - the entire PR context is valuable and relevant [2] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/during-MBlmbcs_5120.recon_alloc.snapshot & https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after- MBlmbcs_5120.recon_alloc.snapshot [3] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/during-MBlmbcs_512.recon_alloc.snapshot & https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after- MBlmbcs_512.recon_alloc.snapshot [4] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/during-MBlmbcs_5120-vs-MBlmbcs_512.png & https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after- MBlmbcs_5120-vs-MBlmbcs_512.png [5] *https://github.com/elixir-lang/elixir/issues/6611#issuecomment-386208496 * [6] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/during-MBas_ageffcaobf-MBlmbcs_512.recon_alloc.snapshot & https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/after-MBas_ageffcaobf-MBlmbcs_512.recon_alloc.snapshot + https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/during-MBas_ageffcbf-MBlmbcs-512.recon_alloc.snapshot & https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/after-MBas_ageffcbf-MBlmbcs-512.recon_alloc.snapshot [7] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/during-MBas_ageffcaobf-MBlmbcs_512-vs- MBas_ageffcbf-MBlmbcs_512.png & https://s3-eu-west-1. amazonaws.com/rabbitmq-share/memory-allocators/after-MBas_ ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png + https://s3-eu-west-1.amazonaws.com/rabbitmq-share/ memory-allocators/during-cpu-MBas_ageffcaobf-MBlmbcs_512- vs-MBas_ageffcbf-MBlmbcs_512.png & https://s3-eu-west-1. amazonaws.com/rabbitmq-share/memory-allocators/during- memory-MBas_ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Thu May 3 12:21:26 2018 From: gerhard@REDACTED (Gerhard Lazu) Date: Thu, 3 May 2018 11:21:26 +0100 Subject: [erlang-questions] How can we increase multiblock carrier utilization for binary_alloc? In-Reply-To: References: Message-ID: Hi Jesper, This is a great explanation, I was able to connect the missing dots. Thank you very much, Gerhard. On Wed, May 2, 2018 at 1:51 PM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > On Tue, May 1, 2018 at 3:06 PM Gerhard Lazu wrote: > >> >> During this exploration, there's a specific thing that's been bugging us: >> why is RSS smaller than the allocated memory? >> >> > This seems fairly obvious to me, but perhaps I am missing something. The > Erlang system has allocated memory from the kernel, but the kernel has not > yet handed that memory out to the process, and hence it is not in the RSS > (Resident Set Size). As you hit new pages, there should be kernel traps, a > page is allocated to the process (bumping RSS) and the program is resumed. > If you allocate a larger carrier the "inner parts" of it might not be > allocated before the first access. > > The other situation is that you have excessive memory pressure and the > system starts removing pages which are possible to remove (they either bear > no data, have already been written to the page/swap file on disk, have > libraries in them, or madvise(2) have been called and told the OS that the > pages are not needed[0]). > > In general, most systems with a GC doesn't give memory back to the OS > straight away. They either keep it indefinitely (perhaps with an madvise(2) > on the areas they don't need), or they have a reaper later on which can > give back fully unused "spans" of memory (Go does this, for instance, but > it isn't instant). > > > [0] Be *very* cautious with madvise(2) since its implementation semantics > are slightly different on Linux/BSD/Illumos. especially around > DONTNEED/FREE and friends, which have different semantics. In particular, > if a don't need a page right now, is the operating system allowed to hand > me a zeroed page later if I start going for that memory again. Bryan > Cantrill has some interesting pointers w.r.t. Lx-branded Illumos Zones and > this :) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From isvilen@REDACTED Thu May 3 12:48:03 2018 From: isvilen@REDACTED (Svilen Ivanov) Date: Thu, 03 May 2018 13:48:03 +0300 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <1525261079.4069.73.camel@ericsson.com> References: <1525261079.4069.73.camel@ericsson.com> Message-ID: <6923ce2d46042004b9dbf153a77a79ff73e30185.camel@applicata.bg> In Kernel User's Guide new logger API chapter "2.6 Example: implement a handler" callback function 'removing_handler' is documented to accept single parameter logger:handler_id(), but in the example for handler that prints to file it takes two parameters (Id and Config). The current implementation seems to call this function with single parameter HandlerId, but that makes not possible to cleanly close the file descriptor as shown in the example. So, removing_handler/1 or removing_handler/2 ? BR, Svilen From essen@REDACTED Thu May 3 13:54:01 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 3 May 2018 13:54:01 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <1525261079.4069.73.camel@ericsson.com> References: <1525261079.4069.73.camel@ericsson.com> Message-ID: <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Hello! Just a quick heads up, having a couple issues: * It doesn't compile on FreeBSD. I can get you more details later but "kerl build git https://github.com/erlang/otp OTP-21.0-rc1 OTP-21.0-rc1" might be a quicker alternative. * SSL is broken. See [1] for example. I can see the same thing happening on 5 different Linux distributions (with different OpenSSL versions) and on OSX. A quick try in the shell is not much better: 1> ssl:start(). ok 2> ssl:connect("google.com", 80, []). {error,{tls_alert,"bad record mac"}} Keep up the good work! [1] https://builds.ninenines.eu/logs/ranch/78/ubuntu/ct_run.ct_ranch@REDACTED/ninenines.ranch.acceptor_SUITE.logs/run.2018-05-03_11.12.52/acceptor_suite.ssl_echo.html On 05/02/2018 01:37 PM, Henrik Nord X wrote: > OTP 21 Release Candidate 1 > > This is the first of two planned release candidates before the OTP 21 > release. The intention whit this release is that you as users try it > and give us feedback if something does not work as expected. Could be a > bug,an unexpected incompatibility, a significant change of > characteristics in a negative direction, etc. > > Erlang/OTP 21 is a new major release with new features, improvements as > well as incompatibilities. > > Potential Incompatibilities > > ?* All Corba applications are now moved from the OTP repository > ????* A new Corba repository will be created https://github.com/erlang > ?* New applications ftp and tftp, moved from inets > ?* ssl no longer supports 3_DES cipher suites or RSA-key exchange > cipher suites by default > ?* erlang:monitor on a primitive node (erl_interface, jinterface, etc) > will no longer fail with badarg exception. Instead a monitor will be > created, but it will only supervise the connection to the node. > > ?Highlights > > ?Erts: > > ????* Enhanced IO scalability > ????* Support for usage of distribution controller processes for > ??????alternative transports, routing etc > ????* compact instructions on 64bit systems for code below 4GB 20% less > ??????memory for loaded code > ????* Rewrite of the efile-driver with NIFs and "Dirty schedulers" > ??????resulting in faster file operations > ????* non-smp VM removed > ????* link and monitor optimized for scalability > ????* os:getenv/putenv now work on thread-safe emulation. No longer in > ??????sync with libc getenv(3). Manual synchronization will be needed. > > Compiler: > ????* Misc compiler optimizations including contributions from the > ??????Elixir team resulting in 10% improvements in benchmarks > ????* "Tuple calls" have been removed from the run-time system. > ????* Code such as f({ok, Val}) -> {ok, Val} is now automatically > ??????rewritten to f({ok, Val} = Tuple) -> Tuple. this reduces code > ??????size, execution time, and removed GC pressure. > ????* More information in stacktrace from a number of operators > ????* erlang:get_stacktrace/0 deprecated to be replaced with try ... > ??????catch C:R:Stacktrace -> ... > ????* Creation of small maps with literal keys optimized. > > Security: > ????* DTLS support in SSL > ????* Enhanced support for distribution over TLS > ????* "unsecure" ciphers removed from defaults in SSL and SSH. > ????* A new option value defined to facilitate implementing exec > ??????servers. Old option kept for compatibility, but now gives errors > ??????on stderror. > > Standard libraries: > ????* New API for logging, logger > ????* New uri_string module for parsing URIs according to "The > ??????standard" > ????* New function lists:search(list,fun/1) -> {ok, Value} | false > ????* Changed default behaviour of .erlang loading. escript, erlc, > ??????dialyzer and typer no longer load an .erlang at all. > > For more details see > http://erlang.org/download/otp_src_21.0-rc1.readme > > Pre built versions for Windows can be fetched here: > http://erlang.org/download/otp_win32_21.0-rc1.exe > http://erlang.org/download/otp_win64_21.0-rc1.exe > > Online documentation can be browsed here: > http://erlang.org/documentation/doc-10.0-rc1/doc/ > > The Erlang/OTP source can also be found at GitHub on the official > Erlang repository, > https://github.com/erlang/otp with tag OTP-21.0-rc1 > > > Thank you for all your contributions! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From essen@REDACTED Thu May 3 14:19:35 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 3 May 2018 14:19:35 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: <484471bc-0045-0a81-60e7-6e39d9e21188@ninenines.eu> On 05/03/2018 01:54 PM, Lo?c Hoguin wrote: > Hello! > > Just a quick heads up, having a couple issues: > > * It doesn't compile on FreeBSD. I can get you more details later but > "kerl build git https://github.com/erlang/otp OTP-21.0-rc1 OTP-21.0-rc1" > might be a quicker alternative. > > * SSL is broken. See [1] for example. I can see the same thing happening > on 5 different Linux distributions (with different OpenSSL versions) and > on OSX. A quick try in the shell is not much better: > > 1> ssl:start(). > ok > 2> ssl:connect("google.com", 80, []). > {error,{tls_alert,"bad record mac"}} That one was a brain fart. This works when the port is correct. > Keep up the good work! > > [1] > https://builds.ninenines.eu/logs/ranch/78/ubuntu/ct_run.ct_ranch@REDACTED/ninenines.ranch.acceptor_SUITE.logs/run.2018-05-03_11.12.52/acceptor_suite.ssl_echo.html > > > On 05/02/2018 01:37 PM, Henrik Nord X wrote: >> OTP 21 Release Candidate 1 >> >> This is the first of two planned release candidates before the OTP 21 >> release. The intention whit this release is that you as users try it >> and give us feedback if something does not work as expected. Could be a >> bug,an unexpected incompatibility, a significant change of >> characteristics in a negative direction, etc. >> >> Erlang/OTP 21 is a new major release with new features, improvements as >> well as incompatibilities. >> >> Potential Incompatibilities >> >> ??* All Corba applications are now moved from the OTP repository >> ?????* A new Corba repository will be created https://github.com/erlang >> ??* New applications ftp and tftp, moved from inets >> ??* ssl no longer supports 3_DES cipher suites or RSA-key exchange >> cipher suites by default >> ??* erlang:monitor on a primitive node (erl_interface, jinterface, etc) >> will no longer fail with badarg exception. Instead a monitor will be >> created, but it will only supervise the connection to the node. >> >> ??Highlights >> >> ??Erts: >> >> ?????* Enhanced IO scalability >> ?????* Support for usage of distribution controller processes for >> ???????alternative transports, routing etc >> ?????* compact instructions on 64bit systems for code below 4GB 20% less >> ???????memory for loaded code >> ?????* Rewrite of the efile-driver with NIFs and "Dirty schedulers" >> ???????resulting in faster file operations >> ?????* non-smp VM removed >> ?????* link and monitor optimized for scalability >> ?????* os:getenv/putenv now work on thread-safe emulation. No longer in >> ???????sync with libc getenv(3). Manual synchronization will be needed. >> >> Compiler: >> ?????* Misc compiler optimizations including contributions from the >> ???????Elixir team resulting in 10% improvements in benchmarks >> ?????* "Tuple calls" have been removed from the run-time system. >> ?????* Code such as f({ok, Val}) -> {ok, Val} is now automatically >> ???????rewritten to f({ok, Val} = Tuple) -> Tuple. this reduces code >> ???????size, execution time, and removed GC pressure. >> ?????* More information in stacktrace from a number of operators >> ?????* erlang:get_stacktrace/0 deprecated to be replaced with try ... >> ???????catch C:R:Stacktrace -> ... >> ?????* Creation of small maps with literal keys optimized. >> >> Security: >> ?????* DTLS support in SSL >> ?????* Enhanced support for distribution over TLS >> ?????* "unsecure" ciphers removed from defaults in SSL and SSH. >> ?????* A new option value defined to facilitate implementing exec >> ???????servers. Old option kept for compatibility, but now gives errors >> ???????on stderror. >> >> Standard libraries: >> ?????* New API for logging, logger >> ?????* New uri_string module for parsing URIs according to "The >> ???????standard" >> ?????* New function lists:search(list,fun/1) -> {ok, Value} | false >> ?????* Changed default behaviour of .erlang loading. escript, erlc, >> ???????dialyzer and typer no longer load an .erlang at all. >> >> For more details see >> http://erlang.org/download/otp_src_21.0-rc1.readme >> >> Pre built versions for Windows can be fetched here: >> http://erlang.org/download/otp_win32_21.0-rc1.exe >> http://erlang.org/download/otp_win64_21.0-rc1.exe >> >> Online documentation can be browsed here: >> http://erlang.org/documentation/doc-10.0-rc1/doc/ >> >> The Erlang/OTP source can also be found at GitHub on the official >> Erlang repository, >> https://github.com/erlang/otp with tag OTP-21.0-rc1 >> >> >> Thank you for all your contributions! >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- Lo?c Hoguin https://ninenines.eu From lukas@REDACTED Thu May 3 15:02:36 2018 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 3 May 2018 15:02:36 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: Hello, On Thu, May 3, 2018 at 1:54 PM, Lo?c Hoguin wrote: > > * It doesn't compile on FreeBSD. I can get you more details later but > "kerl build git https://github.com/erlang/otp OTP-21.0-rc1 OTP-21.0-rc1" > might be a quicker alternative. > Strange, works fine on FreeBSD 10.4 and 11.1 in our tests. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu May 3 15:07:59 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 3 May 2018 16:07:59 +0300 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: Not sure about new logging: $ erl Erlang/OTP 21 [RELEASE CANDIDATE 1] [erts-10.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace] Eshell V10.0 (abort with ^G) 1> whereis(error_logger). undefined 2> From http://erlang.org/documentation/doc-10.0-rc1/lib/kernel-6.0/doc/html/error_logger.html I do not see that error_logger is no more with us. Is it a existing process or it is completely abandoned? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Thu May 3 15:29:52 2018 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 3 May 2018 15:29:52 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: On Thu, May 3, 2018 at 3:07 PM, Max Lapshin wrote: > Not sure about new logging: > > $ erl > > Erlang/OTP 21 [RELEASE CANDIDATE 1] [erts-10.0] [source] [64-bit] > [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace] > > > Eshell V10.0 (abort with ^G) > > 1> whereis(error_logger). > > undefined > > 2> > > > From http://erlang.org/documentation/doc-10.0-rc1/ > lib/kernel-6.0/doc/html/error_logger.html I do not see that error_logger > is no more with us. > > Is it a existing process or it is completely abandoned? > The error_logger event handler is started lazily when the first error_logger:add_report_handler call is done. So the new logging framework does not use the error_logger process, but it is started if legacy code uses it. All calls to error_logger:*_report/msg are redirected to logger which then decides what to do with them. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From zzantozz@REDACTED Thu May 3 15:38:56 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Thu, 03 May 2018 13:38:56 +0000 Subject: [erlang-questions] TLS handshake records don't handle fragments? In-Reply-To: References: Message-ID: Yeah, that looks like it would help. When I was tracing the code, this line was returning zero packets because there wasn't enough data: {Packets, Buf} = tls_handshake:get_tls_handshake(Version,Data,Buf0, Options) ( https://github.com/erlang/otp/commit/1364c7308e17d43d1a2244e3f2bf11cfec3789ef#diff-f053ce3c6b9ffc18f497e4bcbc9661f2R426 ) In that commit, it looks like if no packets can be decoded, you go back and look for more data, which is what it wasn't doing before. On Thu, May 3, 2018 at 1:03 AM Ingela Andin wrote: > Hi Ryan! > > > 2018-05-02 18:08 GMT+02:00 Ryan Stewart : > >> I actually just checked it in the current release of 20, and it works >> there. It's just broken in 18 and 19 that I know of. >> > > > Ah, ok the only relevant commit that I can think of that could have fixed > it is: 1364c7308e17d43d1a2244e3f2bf11cfec3789ef > > Regards Ingela Erlang/OTP team > > > >> >> On Wed, May 2, 2018 at 10:52 AM Ingela Andin >> wrote: >> >>> Hi! >>> >>> I have not had time to check, but it sounds like it could be another bug >>> that I recently fixed. Could you try out the release candidate for OTP 21. >>> >>> Regards Ingela Erlang/OTP Team - Ericsson AB >>> >>> 2018-05-02 16:38 GMT+02:00 Ryan Stewart : >>> >>>> Hi Ingela! Yeah, let me be more clear. The issue is when Erlang is >>>> acting as a client and receives a "certificate request" from a server >>>> during the handshake, as described at >>>> https://tools.ietf.org/html/rfc5246#section-7.4.4. Since Erlang is >>>> receiving the message, it's a decode rather than an encode, but the problem >>>> is that it's not handling fragmented messages, similar to the story I >>>> linked. I'm able to recreate the issue easily by starting a server that >>>> sends certificate requests using the system-installed CAs on a CentOS box, >>>> like this: >>>> >>>> $ openssl req -x509 -newkey rsa:2048 -keyout private.pem -out >>>> public.pem -days 365 -nodes >>>> $ openssl s_server -accept 12345 -cert public.pem -key private.pem >>>> -CAfile /etc/pki/tls/certs/ca-bundle.crt -verify 1 >>>> >>>> Then from Erlang: >>>> >>>> 1> application:ensure_all_started(ssl). >>>> {ok,[crypto,asn1,public_key,ssl]} >>>> 2> ssl:connect("localhost", 12345, []). >>>> >>>> =ERROR REPORT==== 2-May-2018::14:30:13 === >>>> SSL: certify: tls_connection.erl:774:Fatal error: handshake failure >>>> {error,{tls_alert,"handshake failure"}} >>>> >>>> There are so many CAs included by default these days, it's enough to >>>> trigger this bug. If you add a "-msg" to the s_server, you'll see the (very >>>> long) certificate request sent out, followed by a handshake_failure alert >>>> received from the client (Erlang). On the off chance you try this on a >>>> system without enough CAs, I guess we could just generate a bunch of certs >>>> and make our own, big CA file. The certificate request in my test above was >>>> 19185 bytes, which is larger than a TLS record is allowed to be, so it >>>> forces fragmentation. >>>> >>>> On Wed, May 2, 2018 at 5:32 AM Ingela Andin >>>> wrote: >>>> >>>>> Hi! >>>>> >>>>> 2018-05-01 19:23 GMT+02:00 Ryan Stewart : >>>>> >>>>>> I've been getting handshake_failure alerts when trying to connect to >>>>>> a particular server, and I think I've traced it to the fact that the TLS >>>>>> records aren't being handled correctly with respect to fragments. In >>>>>> particular, this server is sending a rather large "certificate request" to >>>>>> allow for client cert auth, and the list is too long to fit in one TLS >>>>>> record. That's breaking the TLS handshake in at least Erlang 18 and 19, I >>>>>> believe. It's basically a mirror image of the problem described in >>>>>> https://bugs.erlang.org/browse/ERL-83. That issue is with Erlang as >>>>>> the TLS server. I'm seeing the same thing with it being the client. Is this >>>>>> addressed somewhere? >>>>>> >>>>>> >>>>> Can you give me a possibility to recreate the issue? That issue you >>>>> described was fixed in 18 and both the client and the server uses the same >>>>> code to encode handshakes. The issue was in the encoding and not in the >>>>> decoding. Can you tell us more details of how you reached your >>>>> conclusion? >>>>> >>>>> Regards Ingela Erlang/OTP Team - Ericsson AB >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>>> >>>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Thu May 3 15:41:28 2018 From: kostis@REDACTED (Kostis Sagonas) Date: Thu, 3 May 2018 15:41:28 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: On 05/03/2018 03:29 PM, Lukas Larsson wrote: > > > On Thu, May 3, 2018 at 3:07 PM, Max Lapshin > wrote: > > Not sure about new logging: > > ... > > From > http://erlang.org/documentation/doc-10.0-rc1/lib/kernel-6.0/doc/html/error_logger.html > > I do not see that error_logger is no more with us. > > Is it a existing process or it is completely abandoned? > > The error_logger event handler is started lazily when the first > error_logger:add_report_handler call is done. So the new logging > framework does not use the error_logger process, but it is started if > legacy code uses it. All calls to error_logger:*_report/msg are > redirected to logger which then decides what to do with them. Something has clearly changed in the behaviour of the error logger in ways that I am not sure I completely understand. I have a simple script to run the HiPE tests which looks as follows: ===================================== #! /bin/sh export ERL_TOP=$PWD export PATH=$ERL_TOP/bin:$PATH TDIR=/tmp/otp_tests rm -rf $TDIR ./otp_build tests $TDIR cd $TDIR ct_run -dir hipe_test ===================================== Till last Friday, this script was running fine, even on master. Since Friday when a lot of changes about the new 'logger' were merged, this script produces tons of lines of the form: =ERROR REPORT==== 3-May-2018::15:38:47.765241 === Error in process <0.1524.0> on node ct@REDACTED with exit value: {timeout_value,[{basic_exceptions,bad_guy,2,[]}]} =WARNING REPORT==== 3-May-2018::15:38:51.663313 === Process <0.2680.0> on node 'ct@REDACTED' enabled access to the emulator internal state. NOTE: This is an erts internal test feature and should *only* be used by OTP test-suites. How does one suppress these? Kostis From garazdawi@REDACTED Thu May 3 15:53:48 2018 From: garazdawi@REDACTED (Lukas Larsson) Date: Thu, 3 May 2018 15:53:48 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: On Thu, May 3, 2018 at 3:41 PM, Kostis Sagonas wrote: > On 05/03/2018 03:29 PM, Lukas Larsson wrote: > >> >> >> On Thu, May 3, 2018 at 3:07 PM, Max Lapshin > > wrote: >> >> Not sure about new logging: >> >> ... >> >> From >> http://erlang.org/documentation/doc-10.0-rc1/lib/kernel-6.0/ >> doc/html/error_logger.html >> > 0/doc/html/error_logger.html> I do not see that error_logger is no >> more with us. >> >> Is it a existing process or it is completely abandoned? >> >> The error_logger event handler is started lazily when the first >> error_logger:add_report_handler call is done. So the new logging >> framework does not use the error_logger process, but it is started if >> legacy code uses it. All calls to error_logger:*_report/msg are redirected >> to logger which then decides what to do with them. >> > > Something has clearly changed in the behaviour of the error logger in ways > that I am not sure I completely understand. > > I have a simple script to run the HiPE tests which looks as follows: > ===================================== > #! /bin/sh > > export ERL_TOP=$PWD > export PATH=$ERL_TOP/bin:$PATH > > TDIR=/tmp/otp_tests > > rm -rf $TDIR > ./otp_build tests $TDIR > cd $TDIR > ct_run -dir hipe_test > ===================================== > > Till last Friday, this script was running fine, even on master. Since > Friday when a lot of changes about the new 'logger' were merged, this > script produces tons of lines of the form: > > =ERROR REPORT==== 3-May-2018::15:38:47.765241 === > Error in process <0.1524.0> on node ct@REDACTED with exit value: > {timeout_value,[{basic_exceptions,bad_guy,2,[]}]} > > =WARNING REPORT==== 3-May-2018::15:38:51.663313 === > Process <0.2680.0> on node 'ct@REDACTED' enabled access to the emulator > internal state. > NOTE: This is an erts internal test feature and should *only* be used by > OTP test-suites. > > > How does one suppress these? There seems to be some bug in the cth_log_redirect. You should be able to do "ct_run -kernel logger_dest silent -dir hipe_test" to work around it for now. We'll fix the bug. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Thu May 3 16:06:19 2018 From: kostis@REDACTED (Kostis Sagonas) Date: Thu, 3 May 2018 16:06:19 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: <27d89d02-b30a-b21a-bb0d-cf263bb59718@cs.ntua.gr> On 05/03/2018 03:53 PM, Lukas Larsson wrote: > > > On Thu, May 3, 2018 at 3:41 PM, Kostis Sagonas > wrote: > > Something has clearly changed in the behaviour of the error logger > in ways that I am not sure I completely understand. > > I have a simple script to run the HiPE tests which looks as follows: > ... > > > How does one suppress these? > > > There seems to be some bug in the cth_log_redirect. You should be able > to do "ct_run -kernel logger_dest silent -dir hipe_test" to work around > it for now. We'll fix the bug. Yep, this works. Thanks for the quick reply! Kostis From max.lapshin@REDACTED Thu May 3 16:28:52 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 3 May 2018 17:28:52 +0300 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: Ok, I was using direct gen_event: gen_event:add_sup_handler(error_logger, Module, Config) On Thu, May 3, 2018 at 4:29 PM, Lukas Larsson wrote: > > > On Thu, May 3, 2018 at 3:07 PM, Max Lapshin wrote: > >> Not sure about new logging: >> >> $ erl >> >> Erlang/OTP 21 [RELEASE CANDIDATE 1] [erts-10.0] [source] [64-bit] >> [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace] >> >> >> Eshell V10.0 (abort with ^G) >> >> 1> whereis(error_logger). >> >> undefined >> >> 2> >> >> >> From http://erlang.org/documentation/doc-10.0-rc1/lib/ >> kernel-6.0/doc/html/error_logger.html I do not see that error_logger is >> no more with us. >> >> Is it a existing process or it is completely abandoned? >> > > The error_logger event handler is started lazily when the first > error_logger:add_report_handler call is done. So the new logging > framework does not use the error_logger process, but it is started if > legacy code uses it. All calls to error_logger:*_report/msg are redirected > to logger which then decides what to do with them. > > Lukas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu May 3 18:08:07 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 3 May 2018 18:08:07 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> Message-ID: <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Hello, On 05/03/2018 01:54 PM, Lo?c Hoguin wrote: > * SSL is broken. See [1] for example. I can see the same thing happening > on 5 different Linux distributions (with different OpenSSL versions) and > on OSX. A quick try in the shell is not much better: OK it's just a very misleading error message I think. Switching my server's test keys from RSA to DSA fixes it so I think this issue is caused by: OTP-14769 Application(s): ssl For security reasons RSA-key exchange cipher suites are no longer supported by default Still, it probably should provide a more helpful error message than this: *** System report during acceptor_SUITE:ssl_echo/1 in ssl 2018-05-03 11:13:04.343 *** =INFO REPORT==== 3-May-2018::11:13:04.342940 === TLS server: In state hello at tls_handshake.erl:130 generated SERVER ALERT: Fatal - Handshake Failure - malformed_handshake_data *** System report during acceptor_SUITE:ssl_echo/1 in ssl 2018-05-03 11:13:04.348 *** =INFO REPORT==== 3-May-2018::11:13:04.348265 === TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure "malformed_handshake_data" sounds like the client would have sent a malformed handshake, ie bad data, when the actual issue seems to be that the certificate configured is no longer supported. The server generating an alert about its own certificate doesn't sound quite right either. That being said I do not really know the intent so I'm guessing a bit. All I know for sure is that it's confusing. Cheers, -- Lo?c Hoguin https://ninenines.eu From bryan.hunt@REDACTED Thu May 3 21:23:37 2018 From: bryan.hunt@REDACTED (Bryan Hunt) Date: Thu, 3 May 2018 20:23:37 +0100 Subject: [erlang-questions] [centos-7]With newer Erlang versions has the eliptic curve crypto situation changed? Message-ID: Hi, Before, when building OTP I always set the compile time option to disable elliptic curve cryptography using the CFLAGS environmental variable (RHEL doesn?t ship with it) : ``` export CFLAGS="-DOPENSSL_NO_EC=1" ./otp_build configure \ --without-odbc \ --without-cosEventDomain \ --without-cosEvent \ --without-cosFileTransfer \ --without-cosNotification \ --without-cosProperty \ --without-cosTime \ --without-cosTransactions \ --without-debugger \ --without-et \ --without-gs \ --without-ic \ --without-javac \ --without-jinterface \ --without-megaco \ --without-observer \ --without-orber \ --without-percept \ --without-typer \ --without-wx \ --without-tv \ --without-diameter \ --without-hipe ``` And that still works for the older versions. But when applied to OTP-21.0-rc1 I receive the following error : ``` gmake[6]: Entering directory `/root/otp/lib/crypto/c_src' CC ../priv/obj/x86_64-unknown-linux-gnu/crypto.o In file included from /usr/include/openssl/ecdh.h:78:0, from /usr/include/openssl/engine.h:86, from crypto.c:63: /usr/include/openssl/ec.h:82:4: error: #error EC is disabled. # error EC is disabled. ^ gmake[6]: Leaving directory `/root/otp/lib/crypto/c_src' gmake[6]: *** [../priv/obj/x86_64-unknown-linux-gnu/crypto.o] Error 1 gmake[5]: *** [release_spec] Error 2 gmake[5]: Leaving directory `/root/otp/lib/crypto/c_src' gmake[4]: *** [release] Error 2 gmake[4]: Leaving directory `/root/otp/lib/crypto/c_src' gmake[3]: *** [release] Error 2 gmake[3]: Leaving directory `/root/otp/lib/crypto/c_src' gmake[2]: *** [release] Error 2 gmake[2]: Leaving directory `/root/otp/lib/crypto' gmake[1]: Leaving directory `/root/otp/lib' gmake[1]: *** [release] Error 2 gmake: *** [release] Error 2 The command '/bin/sh -c ./build-erlang.sh' returned a non-zero code: 1 Unable to find image 'bryanhuntesl/centos7-erlang:OTP-21.0-rc1' locally docker: Error response from daemon: manifest for bryanhuntesl/centos7-erlang:OTP-21.0-rc1 not found. See 'docker run --help?. ``` Has this behaviour changed recently ? Bryan From eric.pailleau@REDACTED Fri May 4 00:43:12 2018 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Fri, 4 May 2018 00:43:12 +0200 Subject: [erlang-questions] [ANN] jason 1.0.2 In-Reply-To: <1525186478.2513256.1356967320.5FDCDEA0@webmail.messagingengine.com> References: <139f0ee3-2170-5b8c-2409-1e5c659a2522@wanadoo.fr> <1525022708.2635058.1354711824.162071BB@webmail.messagingengine.com> <2ee3a4bb-ae78-9876-c717-e79557cbfbe2@wanadoo.fr> <1525186478.2513256.1356967320.5FDCDEA0@webmail.messagingengine.com> Message-ID: Le 01/05/2018 ? 16:54, Tristan Sloughter a ?crit?: > You can publish with rebar3, no need to have a mix.exs. Hi, thanks for the tip, indeed easier. jason is now published under package name jason_erl . Regards From dmcvic@REDACTED Fri May 4 00:55:54 2018 From: dmcvic@REDACTED (Wladimir David Zakrevskyy) Date: Fri, 4 May 2018 00:55:54 +0200 Subject: [erlang-questions] Winternitz signature in erlang? Message-ID: Hello, Winternitz is one quantum safe signature algorithm, exists libraries for this? Thx From ingela.andin@REDACTED Fri May 4 09:32:59 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 4 May 2018 09:32:59 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Message-ID: Hi! 2018-05-03 18:08 GMT+02:00 Lo?c Hoguin : > Hello, > > On 05/03/2018 01:54 PM, Lo?c Hoguin wrote: > >> * SSL is broken. See [1] for example. I can see the same thing happening >> on 5 different Linux distributions (with different OpenSSL versions) and on >> OSX. A quick try in the shell is not much better: >> > > Thank you for shouting, that is what the release candidate is for. So we can catch the problems early! > OK it's just a very misleading error message I think. > Well that depends, this is not really an error message that you should get unless you have a buggy or malicious client. But of course now we might be getting it due to a bug and then it could be misleading! > > Switching my server's test keys from RSA to DSA fixes it so I think this > issue is caused by: > > OTP-14769 Application(s): ssl > > For security reasons RSA-key exchange cipher suites are > no longer supported by default > I do not really suspect this change. RSA-certificates are still supported. Just cipher suites using RSA encryption/decryption in the key exchange process are not supported. When you switched to a DSA-certificate an other cipher suite was picked that did not expose the problem. If there had been no common cipher suites you would have got another error. > > Still, it probably should provide a more helpful error message than this: > > *** System report during acceptor_SUITE:ssl_echo/1 in ssl 2018-05-03 > 11:13:04.343 *** > =INFO REPORT==== 3-May-2018::11:13:04.342940 === > TLS server: In state hello at tls_handshake.erl:130 generated SERVER > ALERT: Fatal - Handshake Failure - malformed_handshake_data > > *** System report during acceptor_SUITE:ssl_echo/1 in ssl 2018-05-03 > 11:13:04.348 *** > =INFO REPORT==== 3-May-2018::11:13:04.348265 === > TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure > > "malformed_handshake_data" sounds like the client would have sent a > malformed handshake, ie bad data, when the actual issue seems to be that > the certificate configured is no longer supported. The server generating an > alert about its own certificate doesn't sound quite right either. > > That being said I do not really know the intent so I'm guessing a bit. All > I know for sure is that it's confusing. > > This error is consistent with one of the errors I am seeing in the nightly builds when running OpenSSL with only default parameters so I suspect something is off in combination version negotiation and cipher suite selection checks. I am looking in to it! Regards Ingela Erlang/OTP Team > Cheers, > > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From heinz@REDACTED Fri May 4 10:05:00 2018 From: heinz@REDACTED (Heinz N. Gies) Date: Fri, 4 May 2018 10:05:00 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Message-ID: I?m having quite some trouble on the SSL front too. I?m giving it a try to move the riak_core code to R21 and the ssl test [1] fails. If I use the original keys (in the repo under test/site1/2-?) I get a handshake failure. Using those keys I get an handshake failure too: openssl req -x509 -newkey rsa:4096 -keyout test/site1-key.pem -out test/site1-cert.pem -days 3650 -nodes -subj '/CN=US' openssl req -x509 -newkey rsa:4096 -keyout test/site2-key.pem -out test/site2-cert.pem -days 3650 -nodes -subj '/CN=US? DSA keys openssl dsaparam -out dsaparams-site1.pem 1024 openssl dsaparam -out dsaparams-site2.pem 1024 openssl req -x509 -newkey dsa:dsaparams-site1.pem -keyout test/site1-key.pem -out test/site1-cert.pem -days 3650 -nodes -subj '/CN=US' openssl req -x509 -newkey dsa:dsaparams-site2.pem -keyout test/site2-key.pem -out test/site2-cert.pem -days 3650 -nodes -subj '/CN=US? Fail with: {badmatch, {error, {options, {keyfile,"test/site1-key.pem", {error, {asn1, {{wrong_tag, {{expected,16}, {got,2, {2, <<74,130,213,43,78,73,39,24,206,62,159, 168,30,65,230,24,14,31,209,192>>}}}}, [{'OTP-PUB-KEY',match_tags,2, [{file,"OTP-PUB-KEY.erl"},{line,20535}]}, {'OTP-PUB-KEY',dec_DSAPrivateKey,2, [{file,"OTP-PUB-KEY.erl"},{line,1789}]}, {'OTP-PUB-KEY',decode,2, [{file,"OTP-PUB-KEY.erl"},{line,1103}]}, {public_key,der_decode,2, [{file,"public_key.erl"},{line,248}]}, {ssl_config,init_private_key,5, [{file,"ssl_config.erl"},{line,114}]}, {ssl_config,init,2, [{file,"ssl_config.erl"},{line,38}]}, {ssl_connection,ssl_config,4, [{file,"ssl_connection.erl"},{line,571}]}, {tls_connection,init,1, [{file,"tls_connection.erl"}, {line,116}]}]}}}}}}} EC keys fail with a handshake failure too openssl ecparam -out ecparams-site1.pem -name prime256v1 openssl ecparam -out ecparams-site2.pem -name prime256v1 openssl req -x509 -newkey ec:ecparams-site1.pem -keyout test/site1-key.pem -out test/site1-cert.pem -days 3650 -nodes -subj '/CN=US' openssl req -x509 -newkey ec:ecparams-site2.pem -keyout test/site2-key.pem -out test/site2-cert.pem -days 3650 -nodes -subj '/CN=US' https://github.com/Kyorai/riak_core/blob/develop/src/riak_core_tcp_mon.erl#L450 > On 4. May 2018, at 09:32, Ingela Andin wrote: > > Hi! > > > 2018-05-03 18:08 GMT+02:00 Lo?c Hoguin >: > Hello, > > On 05/03/2018 01:54 PM, Lo?c Hoguin wrote: > * SSL is broken. See [1] for example. I can see the same thing happening on 5 different Linux distributions (with different OpenSSL versions) and on OSX. A quick try in the shell is not much better: > > > Thank you for shouting, that is what the release candidate is for. So we can catch the problems early! > > > > OK it's just a very misleading error message I think. > > > Well that depends, this is not really an error message that you should get unless you have a buggy or malicious client. But of course now we might be getting it due to a bug and then > it could be misleading! > > > > Switching my server's test keys from RSA to DSA fixes it so I think this issue is caused by: > > OTP-14769 Application(s): ssl > > For security reasons RSA-key exchange cipher suites are > no longer supported by default > > > I do not really suspect this change. RSA-certificates are still supported. Just cipher suites using RSA encryption/decryption in the key exchange process are not supported. > When you switched to a DSA-certificate an other cipher suite was picked that did not expose the problem. If there had been no common cipher suites you would have got another error. > > > > Still, it probably should provide a more helpful error message than this: > > *** System report during acceptor_SUITE:ssl_echo/1 in ssl 2018-05-03 11:13:04.343 *** > =INFO REPORT==== 3-May-2018::11:13:04.342940 === > TLS server: In state hello at tls_handshake.erl:130 generated SERVER ALERT: Fatal - Handshake Failure - malformed_handshake_data > > *** System report during acceptor_SUITE:ssl_echo/1 in ssl 2018-05-03 11:13:04.348 *** > =INFO REPORT==== 3-May-2018::11:13:04.348265 === > TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure > > "malformed_handshake_data" sounds like the client would have sent a malformed handshake, ie bad data, when the actual issue seems to be that the certificate configured is no longer supported. The server generating an alert about its own certificate doesn't sound quite right either. > > That being said I do not really know the intent so I'm guessing a bit. All I know for sure is that it's confusing. > > > > This error is consistent with one of the errors I am seeing in the nightly builds when running OpenSSL with only default parameters so I suspect something is off in combination > version negotiation and cipher suite selection checks. I am looking in to it! > > Regards Ingela Erlang/OTP Team > > > > > > Cheers, > > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From florian.odronitz@REDACTED Fri May 4 10:16:44 2018 From: florian.odronitz@REDACTED (Florian Odronitz) Date: Fri, 4 May 2018 10:16:44 +0200 Subject: [erlang-questions] Memory leak related to "driver_alloc" In-Reply-To: <1525333097.4793.22.camel@ericsson.com> References: <20180502201930.GA10768@jarowit.net> <1525333097.4793.22.camel@ericsson.com> Message-ID: > On 3. May 2018, at 09:38, John H?gberg wrote: > > On Wed, May 02, 2018 at 07:08:34PM +0200, Florian Odronitz wrote: >> When I log into the node I see erlang:memory() reporting 14,6 GB of >> memory basically all attributed to "system". >> Digging deeper recon_alloc:memory(allocated_types) finds that >> basically all the memory is allocated by "driver_alloc". > > If you can try out OTP 21-rc1, the new memory instrumentation features > will help you find the culprit: > > http://blog.erlang.org/Memory-instrumentation-in-OTP-21/ "Those who have used erlang:memory() are probably familiar with how annoyingly general the system category can be. It?s possible to get a bit more information by using erlang:system_info({allocator, Alloc}) but the most it will do is tell you that it?s (say) driver_alloc that eats all that memory and leave you with no clue which one." Thats my situation, exactly! Once Elixir is compatible, I will run one node under OTP 21 and then see if I can get some insight. Thanks for helping Florian PS: In the meanwhile I also looked at memory fragmentation using recon_alloc:fragmentation(current). I found some impressive carriers (>4GB) but they seem to have very good usage. [{{:driver_alloc, 1}, [sbcs_usage: 1.0, mbcs_usage: 0.9999680067349549, sbcs_block_size: 0, sbcs_carriers_size: 0, mbcs_block_size: 4937881312, mbcs_carriers_size: 4938039296]}, {{:driver_alloc, 2}, [sbcs_usage: 1.0, mbcs_usage: 0.9998221050872671, sbcs_block_size: 0, sbcs_carriers_size: 0, mbcs_block_size: 4819741224, mbcs_carriers_size: 4820598784]}, {{:driver_alloc, 3}, [sbcs_usage: 1.0, mbcs_usage: 0.9999122383844911, sbcs_block_size: 0, sbcs_carriers_size: 0, mbcs_block_size: 3822018976, mbcs_carriers_size: 3822354432]}, {{:driver_alloc, 4}, [sbcs_usage: 1.0, mbcs_usage: 0.9967773040787262, sbcs_block_size: 0, sbcs_carriers_size: 0, mbcs_block_size: 1326648648, mbcs_carriers_size: 1330937856]}, {{:driver_alloc, 5}, [sbcs_usage: 1.0, mbcs_usage: 0.9775849991437053, sbcs_block_size: 0, sbcs_carriers_size: 0, mbcs_block_size: 210428096, mbcs_carriers_size: 215252992]}, {{:ll_alloc, 0}, [sbcs_usage: 1.0, mbcs_usage: 0.88204345703125, sbcs_block_size: 0, sbcs_carriers_size: 0, mbcs_block_size: 34683360, mbcs_carriers_size: 39321600]}, > >> I don't fully understand what driver_alloc contains and what could >> drive its growth. >> Any thoughts? > > `driver_alloc` is used for general allocations in drivers and NIFs, > e.g. `enif_alloc()`, `erl_drv_thread_create()`, or > `enif_inspect_iovec()`. > > Regards, > John H?gberg > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Fri May 4 10:58:14 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 04 May 2018 08:58:14 +0000 Subject: [erlang-questions] Memory leak related to "driver_alloc" In-Reply-To: References: <20180502201930.GA10768@jarowit.net> <1525333097.4793.22.camel@ericsson.com> Message-ID: My immediate intuition if you have a 4GB carrier: my goodness, what are you doing to the poor VM to get that? Putting a BluRay disc in memory? I'd definitely try to figure out why that happens, because my intuition has all alarms ringing. On Fri, May 4, 2018 at 10:16 AM Florian Odronitz wrote: > > On 3. May 2018, at 09:38, John H?gberg wrote: > > On Wed, May 02, 2018 at 07:08:34PM +0200, Florian Odronitz wrote: > > When I log into the node I see erlang:memory() reporting 14,6 GB of > memory basically all attributed to "system". > Digging deeper recon_alloc:memory(allocated_types) finds that > basically all the memory is allocated by "driver_alloc". > > > If you can try out OTP 21-rc1, the new memory instrumentation features > will help you find the culprit: > > http://blog.erlang.org/Memory-instrumentation-in-OTP-21/ > > > "Those who have used erlang:memory() are probably familiar with how > annoyingly general the system category can be. It?s possible to get a bit > more information by using erlang:system_info({allocator, Alloc}) but the > most it will do is tell you that it?s (say) driver_alloc that eats all > that memory and leave you with no clue which one." > > Thats my situation, exactly! Once Elixir is compatible, I will run one > node under OTP 21 and then see if I can get some insight. > > Thanks for helping > Florian > > > PS: In the meanwhile I also looked at memory fragmentation > using recon_alloc:fragmentation(current). > I found some impressive carriers (>4GB) but they seem to have very good > usage. > > [{{:driver_alloc, 1}, > [sbcs_usage: 1.0, mbcs_usage: 0.9999680067349549, sbcs_block_size: 0, > sbcs_carriers_size: 0, mbcs_block_size: 4937881312, > mbcs_carriers_size: 4938039296]}, > {{:driver_alloc, 2}, > [sbcs_usage: 1.0, mbcs_usage: 0.9998221050872671, sbcs_block_size: 0, > sbcs_carriers_size: 0, mbcs_block_size: 4819741224, > mbcs_carriers_size: 4820598784]}, > {{:driver_alloc, 3}, > [sbcs_usage: 1.0, mbcs_usage: 0.9999122383844911, sbcs_block_size: 0, > sbcs_carriers_size: 0, mbcs_block_size: 3822018976, > mbcs_carriers_size: 3822354432]}, > {{:driver_alloc, 4}, > [sbcs_usage: 1.0, mbcs_usage: 0.9967773040787262, sbcs_block_size: 0, > sbcs_carriers_size: 0, mbcs_block_size: 1326648648, > mbcs_carriers_size: 1330937856]}, > {{:driver_alloc, 5}, > [sbcs_usage: 1.0, mbcs_usage: 0.9775849991437053, sbcs_block_size: 0, > sbcs_carriers_size: 0, mbcs_block_size: 210428096, > mbcs_carriers_size: 215252992]}, > {{:ll_alloc, 0}, > [sbcs_usage: 1.0, mbcs_usage: 0.88204345703125, sbcs_block_size: 0, > sbcs_carriers_size: 0, mbcs_block_size: 34683360, > mbcs_carriers_size: 39321600]}, > > > > > I don't fully understand what driver_alloc contains and what could > drive its growth. > Any thoughts? > > > `driver_alloc` is used for general allocations in drivers and NIFs, > e.g. `enif_alloc()`, `erl_drv_thread_create()`, or > `enif_inspect_iovec()`. > > Regards, > John H?gberg > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Fri May 4 11:54:29 2018 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Fri, 4 May 2018 11:54:29 +0200 Subject: [erlang-questions] [centos-7]With newer Erlang versions has the eliptic curve crypto situation changed? In-Reply-To: References: Message-ID: <70cf76f9-82aa-cc9c-a7c1-5cfb78289f45@ericsson.com> Thanks! Will be fixed in OTP-21.0-rc2 and in a patch on OTP-20 which also has this error. It is the engine support down in openssl that has problems, so I simply disable our engine support if EC is disabled. /Hans On 05/03/2018 09:23 PM, Bryan Hunt wrote: > > Hi, > > Before, when building OTP I always set the compile time option to disable elliptic curve cryptography using the > CFLAGS environmental variable (RHEL doesn?t ship with it) : > > ``` > export CFLAGS="-DOPENSSL_NO_EC=1" > ./otp_build configure \ > --without-odbc \ > --without-cosEventDomain \ > --without-cosEvent \ > --without-cosFileTransfer \ > --without-cosNotification \ > --without-cosProperty \ > --without-cosTime \ > --without-cosTransactions \ > --without-debugger \ > --without-et \ > --without-gs \ > --without-ic \ > --without-javac \ > --without-jinterface \ > --without-megaco \ > --without-observer \ > --without-orber \ > --without-percept \ > --without-typer \ > --without-wx \ > --without-tv \ > --without-diameter \ > --without-hipe > ``` > > And that still works for the older versions. > > But when applied to OTP-21.0-rc1 I receive the following error : > > ``` > gmake[6]: Entering directory `/root/otp/lib/crypto/c_src' > CC ../priv/obj/x86_64-unknown-linux-gnu/crypto.o > In file included from /usr/include/openssl/ecdh.h:78:0, > from /usr/include/openssl/engine.h:86, > from crypto.c:63: > /usr/include/openssl/ec.h:82:4: error: #error EC is disabled. > # error EC is disabled. > ^ > gmake[6]: Leaving directory `/root/otp/lib/crypto/c_src' > gmake[6]: *** [../priv/obj/x86_64-unknown-linux-gnu/crypto.o] Error 1 > gmake[5]: *** [release_spec] Error 2 > gmake[5]: Leaving directory `/root/otp/lib/crypto/c_src' > gmake[4]: *** [release] Error 2 > gmake[4]: Leaving directory `/root/otp/lib/crypto/c_src' > gmake[3]: *** [release] Error 2 > gmake[3]: Leaving directory `/root/otp/lib/crypto/c_src' > gmake[2]: *** [release] Error 2 > gmake[2]: Leaving directory `/root/otp/lib/crypto' > gmake[1]: Leaving directory `/root/otp/lib' > gmake[1]: *** [release] Error 2 > gmake: *** [release] Error 2 > The command '/bin/sh -c ./build-erlang.sh' returned a non-zero code: 1 > Unable to find image 'bryanhuntesl/centos7-erlang:OTP-21.0-rc1' locally > docker: Error response from daemon: manifest for bryanhuntesl/centos7-erlang:OTP-21.0-rc1 not found. > See 'docker run --help?. > ``` > > Has this behaviour changed recently ? > > Bryan > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From heinz@REDACTED Fri May 4 12:18:43 2018 From: heinz@REDACTED (Heinz N. Gies) Date: Fri, 4 May 2018 12:18:43 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine Message-ID: Since this might get long I want to start with an anecdote. When talking to people about Erlang and why it is great one of the things I mention is stability. I?m not talking about systems not crashing but rather about the fact that change are slow, gentle and breaking changes are rare or nearly unheard off. The example I use is a non trivial toy program I wrote back in 2012 and that it took me a whole of 20 minutes to get code form 2012 working 4 and a half years later (most of the time spent on updating dependencies). Having used other languages and the constant catch up game with changes I always found that impressive and it gives me a warm and fuzzy feeling inside that the code I finish will be finished and don?t require a endless cat and mouse with the language. Even changes that happen like random vs rand are well based with reason - security and performance in that case, and the changes are minor enough that for the most case changing a single function call here and there is enough to even keep complex programs as long as they?re not (ab)using internal knowledge of the implementation (I?m looking at you riak_core). This stability has always be a core value of the Erlang eco system to me, and an important one. While I have the feeling the frequency of deprivations and breaking cross version compatibility has increased over the last releases for most of them I can, as said before, see the reason and the tradeoff made. Now lets talk about senseless murder of gen_fsm. I understand that gen_statem is nicer, and more user friendly, and I don?t want to advocate against it at all, it?s great that we have it and I see forward to useing it for the next state machine I have to write! The next one. I see forward using it for the next one, not the last 100, or even last 15 not even for the last two. I don?t see forward to being forced to go through all libraries that have perfectly fine working gen_fsm without problems or troubles and have to change them. Ripping out a widely used and not broken behaviour just because there is a more fancy one goes against everything I value in erlang as an eco system. It forces developers to do the kind of busy work I detest in other platforms. Worst of all forcing people to fundamentally re-write perfectly good code means bugs, bugs we wouldn?t have had if we would just have left the code alone. Bugs that are probably non obvious as differences are subtile. And in the end for absolutely no benefit. Now it?s just a rant without a proposal. And I detest whining without constructive criticism as much as I detest platforms forcing changes that are not required. So here is what I?d like to suggest: - Keep gem_fsm around, it has been a friend to many for a long time and there is a lot of code using it that shouldn?t have to change to use newer erlang versions. - Encourage people to use gen_statm for new code, and if it is really that much better then gen_fsm that it?d warrant to rip it out that should happen on it?s own. Heinz - member of team gen_fsm ?? PS: I?m not just writing this out of no-where I?ve been fighting subtile bugs on gen_fsm -> gem_statem code that was ported by people a lot smarter then me and I see no chance in hell that if they can?t pull that off on simple code I will ever be able to do that on complex code. PPS: if the this plea falls on deaf ears (which I really hope it does?t) here is a plan-b for team gen_fsm https://gitlab.com/Project-FiFo/gen_fsm_compat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From roger@REDACTED Fri May 4 12:25:19 2018 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 4 May 2018 11:25:19 +0100 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Message-ID: On 4 May 2018 at 08:32, Ingela Andin wrote: > This error is consistent with one of the errors I am seeing in the nightly > builds when running OpenSSL with only default parameters so I suspect > something is off in combination > version negotiation and cipher suite selection checks. I am looking in to > it! I'm seeing the same, if it helps to reproduce. I generated my certificates with: #!/bin/sh # Create the CA key and certificate. openssl genrsa -out ca.key 2048 openssl req -new -x509 -nodes -key ca.key -days 3653 -out ca.pem -subj "/CN=Test CA" # Create the server key and CSR. openssl genrsa -out server.key 2048 openssl req -new -key server.key -out server.csr -subj "/CN=localhost" # Sign the CSR with the CA key. serial=$(date +"%s") openssl x509 -req -days 3563 -CA ca.pem -CAserial $serial -CAcreateserial -CAkey ca.key -in server.csr -out server.pem rm $serial I tested with: % Server {ok, _} = application:ensure_all_started(ssl). Port = 11002. LOpts = [{certfile, "server.pem"}, {keyfile, "server.key"}]. {ok, LSock} = ssl:listen(Port, LOpts). {ok, CSock} = ssl:transport_accept(LSock). ok = ssl:ssl_accept(CSock). % Client {ok, _} = application:ensure_all_started(ssl). Port = 11002. COpts = [{verify, verify_peer}, {cacertfile, "ca.pem"}]. {ok, Sock} = ssl:connect("localhost", Port, COpts). The server reports: =INFO REPORT==== 4-May-2018::11:22:20.971130 === TLS server: In state hello at tls_handshake.erl:130 generated SERVER ALERT: Fatal - Handshake Failure - malformed_handshake_data ** exception error: no match of right hand side value {error,{tls_alert,"handshake failure"}} The client reports: =INFO REPORT==== 4-May-2018::11:22:20.981524 === TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure ** exception error: no match of right hand side value {error,{tls_alert,"handshake failure"}} The same code works fine with 20.3.1 Thanks, Roger. From bryan.hunt@REDACTED Fri May 4 12:28:33 2018 From: bryan.hunt@REDACTED (Bryan Hunt) Date: Fri, 4 May 2018 11:28:33 +0100 Subject: [erlang-questions] [centos-7]With newer Erlang versions has the eliptic curve crypto situation changed? In-Reply-To: <70cf76f9-82aa-cc9c-a7c1-5cfb78289f45@ericsson.com> References: <70cf76f9-82aa-cc9c-a7c1-5cfb78289f45@ericsson.com> Message-ID: <87D2AD9D-C31A-4F36-8AE6-4C898CF86AAA@erlang-solutions.com> Great, thanks for the quick response Hans. B > On 4 May 2018, at 10:54, Hans Nilsson R wrote: > > Thanks! > > Will be fixed in OTP-21.0-rc2 and in a patch on OTP-20 which also has this error. > > It is the engine support down in openssl that has problems, so I simply disable our engine support if EC is disabled. > > /Hans > > On 05/03/2018 09:23 PM, Bryan Hunt wrote: >> >> Hi, >> >> Before, when building OTP I always set the compile time option to disable elliptic curve cryptography using the >> CFLAGS environmental variable (RHEL doesn?t ship with it) : >> >> ``` >> export CFLAGS="-DOPENSSL_NO_EC=1" >> ./otp_build configure \ >> --without-odbc \ >> --without-cosEventDomain \ >> --without-cosEvent \ >> --without-cosFileTransfer \ >> --without-cosNotification \ >> --without-cosProperty \ >> --without-cosTime \ >> --without-cosTransactions \ >> --without-debugger \ >> --without-et \ >> --without-gs \ >> --without-ic \ >> --without-javac \ >> --without-jinterface \ >> --without-megaco \ >> --without-observer \ >> --without-orber \ >> --without-percept \ >> --without-typer \ >> --without-wx \ >> --without-tv \ >> --without-diameter \ >> --without-hipe >> ``` >> >> And that still works for the older versions. >> >> But when applied to OTP-21.0-rc1 I receive the following error : >> >> ``` >> gmake[6]: Entering directory `/root/otp/lib/crypto/c_src' >> CC ../priv/obj/x86_64-unknown-linux-gnu/crypto.o >> In file included from /usr/include/openssl/ecdh.h:78:0, >> from /usr/include/openssl/engine.h:86, >> from crypto.c:63: >> /usr/include/openssl/ec.h:82:4: error: #error EC is disabled. >> # error EC is disabled. >> ^ >> gmake[6]: Leaving directory `/root/otp/lib/crypto/c_src' >> gmake[6]: *** [../priv/obj/x86_64-unknown-linux-gnu/crypto.o] Error 1 >> gmake[5]: *** [release_spec] Error 2 >> gmake[5]: Leaving directory `/root/otp/lib/crypto/c_src' >> gmake[4]: *** [release] Error 2 >> gmake[4]: Leaving directory `/root/otp/lib/crypto/c_src' >> gmake[3]: *** [release] Error 2 >> gmake[3]: Leaving directory `/root/otp/lib/crypto/c_src' >> gmake[2]: *** [release] Error 2 >> gmake[2]: Leaving directory `/root/otp/lib/crypto' >> gmake[1]: Leaving directory `/root/otp/lib' >> gmake[1]: *** [release] Error 2 >> gmake: *** [release] Error 2 >> The command '/bin/sh -c ./build-erlang.sh' returned a non-zero code: 1 >> Unable to find image 'bryanhuntesl/centos7-erlang:OTP-21.0-rc1' locally >> docker: Error response from daemon: manifest for bryanhuntesl/centos7-erlang:OTP-21.0-rc1 not found. >> See 'docker run --help?. >> ``` >> >> Has this behaviour changed recently ? >> >> Bryan >> >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> From ingela.andin@REDACTED Fri May 4 12:36:29 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 4 May 2018 12:36:29 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Message-ID: Hi! I traced my problem to sslv2_hello compatibility. I will give you something to try later today! Regards Ingela Erlang/OTP team 2018-05-04 10:05 GMT+02:00 Heinz N. Gies : > I?m having quite some trouble on the SSL front too. I?m giving it a try to > move the riak_core code to R21 and the ssl test [1] fails. > > If I use the original keys (in the repo under test/site1/2-?) I get a > handshake failure. > > Using those keys I get an handshake failure too: > > openssl req -x509 -newkey rsa:4096 -keyout test/site1-key.pem -out > test/site1-cert.pem -days 3650 -nodes -subj '/CN=US' > openssl req -x509 -newkey rsa:4096 -keyout test/site2-key.pem -out > test/site2-cert.pem -days 3650 -nodes -subj '/CN=US? > > DSA keys > > openssl dsaparam -out dsaparams-site1.pem 1024 > openssl dsaparam -out dsaparams-site2.pem 1024 > openssl req -x509 -newkey dsa:dsaparams-site1.pem -keyout > test/site1-key.pem -out test/site1-cert.pem -days 3650 -nodes -subj '/CN=US' > openssl req -x509 -newkey dsa:dsaparams-site2.pem -keyout > test/site2-key.pem -out test/site2-cert.pem -days 3650 -nodes -subj '/CN=US? > > Fail with: > > {badmatch, > {error, > {options, > {keyfile,"test/site1-key.pem", > {error, > {asn1, > {{wrong_tag, > {{expected,16}, > {got,2, > {2, > <<74,130,213,43,78,73,39,24, > 206,62,159, > 168,30,65,230,24,14,31,209, > 192>>}}}}, > [{'OTP-PUB-KEY',match_tags,2, > [{file,"OTP-PUB-KEY.erl"},{line,20535}]}, > {'OTP-PUB-KEY',dec_DSAPrivateKey,2, > [{file,"OTP-PUB-KEY.erl"},{line,1789}]}, > {'OTP-PUB-KEY',decode,2, > [{file,"OTP-PUB-KEY.erl"},{line,1103}]}, > {public_key,der_decode,2, > [{file,"public_key.erl"},{line,248}]}, > {ssl_config,init_private_key,5, > [{file,"ssl_config.erl"},{line,114}]}, > {ssl_config,init,2, > [{file,"ssl_config.erl"},{line,38}]}, > {ssl_connection,ssl_config,4, > [{file,"ssl_connection.erl"},{line,571}]}, > {tls_connection,init,1, > [{file,"tls_connection.erl"}, > {line,116}]}]}}}}}}} > > > EC keys fail with a handshake failure too > > openssl ecparam -out ecparams-site1.pem -name prime256v1 > openssl ecparam -out ecparams-site2.pem -name prime256v1 > openssl req -x509 -newkey ec:ecparams-site1.pem -keyout test/site1-key.pem > -out test/site1-cert.pem -days 3650 -nodes -subj '/CN=US' > openssl req -x509 -newkey ec:ecparams-site2.pem -keyout test/site2-key.pem > -out test/site2-cert.pem -days 3650 -nodes -subj '/CN=US' > > > https://github.com/Kyorai/riak_core/blob/develop/src/ > riak_core_tcp_mon.erl#L450 > > > > > On 4. May 2018, at 09:32, Ingela Andin wrote: > > Hi! > > > 2018-05-03 18:08 GMT+02:00 Lo?c Hoguin : > >> Hello, >> >> On 05/03/2018 01:54 PM, Lo?c Hoguin wrote: >> >>> * SSL is broken. See [1] for example. I can see the same thing happening >>> on 5 different Linux distributions (with different OpenSSL versions) and on >>> OSX. A quick try in the shell is not much better: >>> >> >> > Thank you for shouting, that is what the release candidate is for. So we > can catch the problems early! > > > > >> OK it's just a very misleading error message I think. >> > > > Well that depends, this is not really an error message that you should get > unless you have a buggy or malicious client. But of course now we might be > getting it due to a bug and then > it could be misleading! > > > >> >> Switching my server's test keys from RSA to DSA fixes it so I think this >> issue is caused by: >> >> OTP-14769 Application(s): ssl >> >> For security reasons RSA-key exchange cipher suites are >> no longer supported by default >> > > > I do not really suspect this change. RSA-certificates are still supported. > Just cipher suites using RSA encryption/decryption in the key exchange > process are not supported. > When you switched to a DSA-certificate an other cipher suite was picked > that did not expose the problem. If there had been no common cipher suites > you would have got another error. > > > >> >> Still, it probably should provide a more helpful error message than this: >> >> *** System report during acceptor_SUITE:ssl_echo/1 in ssl 2018-05-03 >> 11:13:04.343 *** >> =INFO REPORT==== 3-May-2018::11:13:04.342940 === >> TLS server: In state hello at tls_handshake.erl:130 generated SERVER >> ALERT: Fatal - Handshake Failure - malformed_handshake_data >> >> *** System report during acceptor_SUITE:ssl_echo/1 in ssl 2018-05-03 >> 11:13:04.348 *** >> =INFO REPORT==== 3-May-2018::11:13:04.348265 === >> TLS client: In state hello received SERVER ALERT: Fatal - Handshake >> Failure >> >> "malformed_handshake_data" sounds like the client would have sent a >> malformed handshake, ie bad data, when the actual issue seems to be that >> the certificate configured is no longer supported. The server generating an >> alert about its own certificate doesn't sound quite right either. >> >> That being said I do not really know the intent so I'm guessing a bit. >> All I know for sure is that it's confusing. >> >> > > This error is consistent with one of the errors I am seeing in the nightly > builds when running OpenSSL with only default parameters so I suspect > something is off in combination > version negotiation and cipher suite selection checks. I am looking in to > it! > > Regards Ingela Erlang/OTP Team > > > > > > >> Cheers, >> >> >> -- >> Lo?c Hoguin >> https://ninenines.eu >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri May 4 12:36:50 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Fri, 4 May 2018 12:36:50 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: <2a7ad01a-da5f-c239-fe2d-d2c2ba0c6607@ninenines.eu> What's the issue exactly? Is there plans to remove gen_fsm already, or is it just about its deprecation? On 05/04/2018 12:18 PM, Heinz N. Gies wrote: > Since this might get long I want to start with an anecdote. > > When talking to people about Erlang and why it is great one of the > things I mention is stability. I?m not talking about systems not > crashing but rather about the fact that change are slow, gentle and > breaking changes are rare or nearly unheard off. The example I use is a > non trivial toy program I wrote back in 2012 and that it took me a whole > of 20 minutes to get code form 2012 working 4 and a half years later > ?(most of the time spent on updating dependencies). > > Having used other languages and the constant catch up game with changes > I always found that impressive ?and it gives me a warm and fuzzy feeling > inside that the code I finish will be finished and don?t require a > endless cat and mouse with the language. > > Even changes that happen like random vs rand are well based with reason > - security and performance in that case, and the changes are minor > enough that for the most case changing a single function call here and > there is enough to even keep complex programs as long as they?re not > (ab)using internal knowledge of the implementation (I?m looking at you > riak_core). > > This stability has always be a core value of the Erlang eco system to > me, and an important one. > > While I have the feeling the frequency of deprivations and breaking > cross version compatibility has increased over the last releases for > most of them I can, as said before, see the reason and the tradeoff made. > > Now lets talk about senseless murder of gen_fsm. > > I understand that gen_statem is nicer, and more user friendly, and I > don?t want to advocate against it at all, it?s great that we have it and > I see forward to useing it for the next state machine I have to write! > > The *next* one. I see forward using it *for the next one*, not the last > 100, or even last 15 not even for the last two. I don?t see forward to > being forced to go through all libraries that have perfectly fine > working gen_fsm without problems or troubles and have to change them. > > Ripping out a widely used and not broken behaviour just because there is > a more fancy one goes against everything I value in erlang as an eco > system. It forces developers to do the kind of busy work I detest in > other platforms. > > Worst of all forcing people to fundamentally re-write perfectly good > code means bugs, bugs we wouldn?t have had if we would just have left > the code alone. Bugs that are probably non obvious as differences are > subtile. And in the end for absolutely no benefit. > > Now it?s just a rant without a proposal. And I detest whining without > constructive criticism as much as I detest platforms forcing changes > that are not required. So here is what I?d like to suggest: > > - Keep gem_fsm around, it has been a friend to many for a long time and > there is a lot of code using it that shouldn?t have to change to use > newer erlang versions. > - Encourage people to use gen_statm for new code, and if it is really > that much better then gen_fsm that it?d warrant to rip it out that > should happen on it?s own. > > Heinz - member of team gen_fsm ?? > > PS: I?m not just writing this out of no-where I?ve been fighting subtile > bugs on gen_fsm -> gem_statem code that was ported by people a lot > smarter then me and I see no chance in hell that if they can?t pull that > off on simple code I will ever be able to do that on complex code. > > PPS: if the this plea falls on deaf ears (which I really hope it does?t) > here is a plan-b for team gen_fsm > https://gitlab.com/Project-FiFo/gen_fsm_compat > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From heinz@REDACTED Fri May 4 12:54:53 2018 From: heinz@REDACTED (Heinz N. Gies) Date: Fri, 4 May 2018 12:54:53 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: <935c1dd5-bbcd-c450-6fc2-9f61032b0107@ninenines.eu> References: <2a7ad01a-da5f-c239-fe2d-d2c2ba0c6607@ninenines.eu> <935c1dd5-bbcd-c450-6fc2-9f61032b0107@ninenines.eu> Message-ID: Oops I forget to reply all again :/ Welp. Yes it?s early. But I honestly don?t see much reason to ever deprecate it and force people to change working code. Perhaps there is a good reason other then ?gen_statm is nicer? I just don?t see. > On 4. May 2018, at 12:48, Lo?c Hoguin wrote: > > Ah I was looking in the wrong places... > > Yeah that's a bit early. > > On 05/04/2018 12:45 PM, Heinz N. Gies wrote: >> The code says it?s going to be removed in the next major release: >> https://github.com/erlang/otp/blob/master/lib/stdlib/src/gen_fsm.erl#L132 >> The problem I encountered was very odd edge case of different behaviour on shut down, no idea why I killed it with fire and went back to gen_fsm. It?s subtle, didn?t show up in any tests for the lib itself until it ended up in a bigger project where things got out of hand. >>> On 4. May 2018, at 12:36, Lo?c Hoguin > wrote: >>> >>> What's the issue exactly? Is there plans to remove gen_fsm already, or is it just about its deprecation? >>> >>> On 05/04/2018 12:18 PM, Heinz N. Gies wrote: >>>> Since this might get long I want to start with an anecdote. >>>> When talking to people about Erlang and why it is great one of the things I mention is stability. I?m not talking about systems not crashing but rather about the fact that change are slow, gentle and breaking changes are rare or nearly unheard off. The example I use is a non trivial toy program I wrote back in 2012 and that it took me a whole of 20 minutes to get code form 2012 working 4 and a half years later (most of the time spent on updating dependencies). >>>> Having used other languages and the constant catch up game with changes I always found that impressive and it gives me a warm and fuzzy feeling inside that the code I finish will be finished and don?t require a endless cat and mouse with the language. >>>> Even changes that happen like random vs rand are well based with reason - security and performance in that case, and the changes are minor enough that for the most case changing a single function call here and there is enough to even keep complex programs as long as they?re not (ab)using internal knowledge of the implementation (I?m looking at you riak_core). >>>> This stability has always be a core value of the Erlang eco system to me, and an important one. >>>> While I have the feeling the frequency of deprivations and breaking cross version compatibility has increased over the last releases for most of them I can, as said before, see the reason and the tradeoff made. >>>> Now lets talk about senseless murder of gen_fsm. >>>> I understand that gen_statem is nicer, and more user friendly, and I don?t want to advocate against it at all, it?s great that we have it and I see forward to useing it for the next state machine I have to write! >>>> The *next* one. I see forward using it *for the next one*, not the last 100, or even last 15 not even for the last two. I don?t see forward to being forced to go through all libraries that have perfectly fine working gen_fsm without problems or troubles and have to change them. >>>> Ripping out a widely used and not broken behaviour just because there is a more fancy one goes against everything I value in erlang as an eco system. It forces developers to do the kind of busy work I detest in other platforms. >>>> Worst of all forcing people to fundamentally re-write perfectly good code means bugs, bugs we wouldn?t have had if we would just have left the code alone. Bugs that are probably non obvious as differences are subtile. And in the end for absolutely no benefit. >>>> Now it?s just a rant without a proposal. And I detest whining without constructive criticism as much as I detest platforms forcing changes that are not required. So here is what I?d like to suggest: >>>> - Keep gem_fsm around, it has been a friend to many for a long time and there is a lot of code using it that shouldn?t have to change to use newer erlang versions. >>>> - Encourage people to use gen_statm for new code, and if it is really that much better then gen_fsm that it?d warrant to rip it out that should happen on it?s own. >>>> Heinz - member of team gen_fsm ?? >>>> PS: I?m not just writing this out of no-where I?ve been fighting subtile bugs on gen_fsm -> gem_statem code that was ported by people a lot smarter then me and I see no chance in hell that if they can?t pull that off on simple code I will ever be able to do that on complex code. >>>> PPS: if the this plea falls on deaf ears (which I really hope it does?t) here is a plan-b for team gen_fsm https://gitlab.com/Project-FiFo/gen_fsm_compat >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> -- >>> Lo?c Hoguin >>> https://ninenines.eu > > -- > Lo?c Hoguin > https://ninenines.eu -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From k.petrauskas@REDACTED Fri May 4 13:07:59 2018 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Fri, 4 May 2018 14:07:59 +0300 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: Hi, I agree with Heinz. The deprecation of gen_fsm forced me to stuck on OTP 19 and I dont see any reasonable path to upgrade. There are a lot of dependencies using gen_fsm and I don't see, how one can make code be compatible between releases not providing gen_statem and deprecating gen_fsm. With simple functions a macro approach is good enough, although it does not work with the gen_fsm. The deprecation of the erlang:get_stacktrace() feels similarly. Karolis On Fri, May 4, 2018 at 1:18 PM, Heinz N. Gies wrote: > Since this might get long I want to start with an anecdote. > > When talking to people about Erlang and why it is great one of the things > I mention is stability. I?m not talking about systems not crashing but > rather about the fact that change are slow, gentle and breaking changes are > rare or nearly unheard off. The example I use is a non trivial toy program > I wrote back in 2012 and that it took me a whole of 20 minutes to get code > form 2012 working 4 and a half years later (most of the time spent on > updating dependencies). > > Having used other languages and the constant catch up game with changes I > always found that impressive and it gives me a warm and fuzzy feeling > inside that the code I finish will be finished and don?t require a endless > cat and mouse with the language. > > Even changes that happen like random vs rand are well based with reason - > security and performance in that case, and the changes are minor enough > that for the most case changing a single function call here and there is > enough to even keep complex programs as long as they?re not (ab)using > internal knowledge of the implementation (I?m looking at you riak_core). > > This stability has always be a core value of the Erlang eco system to me, > and an important one. > > While I have the feeling the frequency of deprivations and breaking cross > version compatibility has increased over the last releases for most of them > I can, as said before, see the reason and the tradeoff made. > > Now lets talk about senseless murder of gen_fsm. > > I understand that gen_statem is nicer, and more user friendly, and I don?t > want to advocate against it at all, it?s great that we have it and I see > forward to useing it for the next state machine I have to write! > > The *next* one. I see forward using it *for the next one*, not the last > 100, or even last 15 not even for the last two. I don?t see forward to > being forced to go through all libraries that have perfectly fine working > gen_fsm without problems or troubles and have to change them. > > Ripping out a widely used and not broken behaviour just because there is a > more fancy one goes against everything I value in erlang as an eco system. > It forces developers to do the kind of busy work I detest in other > platforms. > > Worst of all forcing people to fundamentally re-write perfectly good code > means bugs, bugs we wouldn?t have had if we would just have left the code > alone. Bugs that are probably non obvious as differences are subtile. And > in the end for absolutely no benefit. > > Now it?s just a rant without a proposal. And I detest whining without > constructive criticism as much as I detest platforms forcing changes that > are not required. So here is what I?d like to suggest: > > - Keep gem_fsm around, it has been a friend to many for a long time and > there is a lot of code using it that shouldn?t have to change to use newer > erlang versions. > - Encourage people to use gen_statm for new code, and if it is really that > much better then gen_fsm that it?d warrant to rip it out that should happen > on it?s own. > > Heinz - member of team gen_fsm ?? > > PS: I?m not just writing this out of no-where I?ve been fighting subtile > bugs on gen_fsm -> gem_statem code that was ported by people a lot smarter > then me and I see no chance in hell that if they can?t pull that off on > simple code I will ever be able to do that on complex code. > > PPS: if the this plea falls on deaf ears (which I really hope it does?t) > here is a plan-b for team gen_fsm https://gitlab.com/ > Project-FiFo/gen_fsm_compat > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Fri May 4 13:58:40 2018 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 4 May 2018 12:58:40 +0100 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: On 4 May 2018 at 12:07, Karolis Petrauskas wrote: > I agree with Heinz. The deprecation of gen_fsm forced me to stuck on OTP > 19 and I dont see any reasonable path to upgrade. > We upgraded successfully to OTP 20 (and compiled against OTP 21) by simply adding the following to the top of our gen_fsm modules: -compile([nowarn_deprecated_function]). % gen_fsm is deprecated; use gen_statem It'll be a problem when gen_fsm is *removed*, but until then, we'll stick with this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From heinz@REDACTED Fri May 4 14:03:11 2018 From: heinz@REDACTED (Heinz N. Gies) Date: Fri, 4 May 2018 14:03:11 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> I have thought of that but I don?t think that?s a real option. Deprication warnings are important, this would turn all of them off which doesn?t help code quality. I want to know if something is deprecated and fix it in most cases. But as Karolis pointed out, it?s easy for a function or two but for a whole behaviour it?s nearly impossible. It basically creates two erlang worlds, pre-statem and post-statem. The stack trace is not great but it?s not as bad, it?s somewhat possible to just wrap the whole function calling it into a ifdef and call it a day. Not great it still is a decent amount of code duplication but it?s possible. https://github.com/Kyorai/clique/commit/b0e89de3cbb56396b59a1023903c7259ec2ef145#diff-7a19efbe2afedfb64aa7ec6690e9a13e would be an example. It?s more annoying then other changes but not nearly as bad as the gen_fsm > On 4. May 2018, at 13:58, Roger Lipscombe wrote: > > On 4 May 2018 at 12:07, Karolis Petrauskas > wrote: > I agree with Heinz. The deprecation of gen_fsm forced me to stuck on OTP 19 and I dont see any reasonable path to upgrade. > > We upgraded successfully to OTP 20 (and compiled against OTP 21) by simply adding the following to the top of our gen_fsm modules: > > -compile([nowarn_deprecated_function]). % gen_fsm is deprecated; use gen_statem > > It'll be a problem when gen_fsm is *removed*, but until then, we'll stick with this. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From oliver.korpilla@REDACTED Fri May 4 14:12:36 2018 From: oliver.korpilla@REDACTED (Oliver Korpilla) Date: Fri, 04 May 2018 08:12:36 -0400 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> Message-ID: Except for the init bug in 19.0 (which our infrastructure was fixed on for a long time) we found migrating to gen_statem a painless experience with decent improvements. But maybe we simply use a different or limited feature set? Oliver On May 4, 2018 8:03:11 AM EDT, "Heinz N. Gies" wrote: >I have thought of that but I don?t think that?s a real option. >Deprication warnings are important, this would turn all of them off >which doesn?t help code quality. I want to know if something is >deprecated and fix it in most cases. But as Karolis pointed out, it?s >easy for a function or two but for a whole behaviour it?s nearly >impossible. > >It basically creates two erlang worlds, pre-statem and post-statem. > >The stack trace is not great but it?s not as bad, it?s somewhat >possible to just wrap the whole function calling it into a ifdef and >call it a day. Not great it still is a decent amount of code >duplication but it?s possible. >https://github.com/Kyorai/clique/commit/b0e89de3cbb56396b59a1023903c7259ec2ef145#diff-7a19efbe2afedfb64aa7ec6690e9a13e > >would be an example. It?s more annoying then other changes but not >nearly as bad as the gen_fsm > >> On 4. May 2018, at 13:58, Roger Lipscombe >wrote: >> >> On 4 May 2018 at 12:07, Karolis Petrauskas > wrote: >> I agree with Heinz. The deprecation of gen_fsm forced me to stuck on >OTP 19 and I dont see any reasonable path to upgrade. >> >> We upgraded successfully to OTP 20 (and compiled against OTP 21) by >simply adding the following to the top of our gen_fsm modules: >> >> -compile([nowarn_deprecated_function]). % gen_fsm is deprecated; use >gen_statem >> >> It'll be a problem when gen_fsm is *removed*, but until then, we'll >stick with this. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From getonga@REDACTED Fri May 4 09:51:45 2018 From: getonga@REDACTED (=?utf-8?B?Z2V0b25nYQ==?=) Date: Fri, 4 May 2018 15:51:45 +0800 Subject: [erlang-questions] gen_statem is blocking in gen do_call, a lot of messages does not be handled. Message-ID: The process info is ``` process_info(pid(0, 818, 0)). [{current_function,{gen,do_call,4}}, {initial_call,{proc_lib,init_p,5}}, {status,waiting}, {message_queue_len,10}, {messages,[1,2,3,3,4,5,6,7,8,9,10]}, {links,[<0.514.0>,<0.1294.0>]}, {dictionary,[{rand_seed,{#{bits => 58,jump => #Fun, next => #Fun,type => exrop, uniform => #Fun, uniform_n => #Fun,weak_low_bits => 1}, [218965415455180470|167242355698640157]}}, {'$initial_call',{time_server,init,1}}, {'$ancestors',[time_sup,app_sup,<0.512.0>]}]}, {trap_exit,true}, {error_handler,error_handler}, {priority,normal}, {group_leader,<0.511.0>}, {total_heap_size,987}, {heap_size,987}, {stack_size,47}, {reductions,27674}, {garbage_collection,[{max_heap_size,#{error_logger => true,kill => true,size => 0}}, {min_bin_vheap_size,46422}, {min_heap_size,233}, {fullsweep_after,65535}, {minor_gcs,0}]}, {suspending,[]}] ``` and I look into the gen_statem.erl code, the gen:call/4 is running without timeout: ``` call_dirty(ServerRef, Request, Timeout, T) -> try gen:call(ServerRef, '$gen_call', Request, T) of {ok,Reply} -> Reply catch Class:Reason -> erlang:raise( Class, {Reason,{?MODULE,call,[ServerRef,Request,Timeout]}}, erlang:get_stacktrace()) end. ``` Is the code blocks my gen_statem process ? How can I disable it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From heinz@REDACTED Fri May 4 14:25:35 2018 From: heinz@REDACTED (Heinz N. Gies) Date: Fri, 4 May 2018 14:25:35 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> Message-ID: Hi Oliver, I am not advocating to forbid migrating code to gen_statem when the maintainer feels it is an improvement for their code. I?m advocating for not forcing maintainers doing so for no reason. Karolis brought up another very good point I didn?t even think about. Code written with gen_statem will not be able to run on older erlang releases. While that?s probably not a issue on your own code where you control the releases completely it?s a nightmare for open source and libraries. > On 4. May 2018, at 14:12, Oliver Korpilla wrote: > > Except for the init bug in 19.0 (which our infrastructure was fixed on for a long time) we found migrating to gen_statem a painless experience with decent improvements. But maybe we simply use a different or limited feature set? > > Oliver > > On May 4, 2018 8:03:11 AM EDT, "Heinz N. Gies" wrote: > I have thought of that but I don?t think that?s a real option. Deprication warnings are important, this would turn all of them off which doesn?t help code quality. I want to know if something is deprecated and fix it in most cases. But as Karolis pointed out, it?s easy for a function or two but for a whole behaviour it?s nearly impossible. > > It basically creates two erlang worlds, pre-statem and post-statem. > > The stack trace is not great but it?s not as bad, it?s somewhat possible to just wrap the whole function calling it into a ifdef and call it a day. Not great it still is a decent amount of code duplication but it?s possible. https://github.com/Kyorai/clique/commit/b0e89de3cbb56396b59a1023903c7259ec2ef145#diff-7a19efbe2afedfb64aa7ec6690e9a13e would be an example. It?s more annoying then other changes but not nearly as bad as the gen_fsm > >> On 4. May 2018, at 13:58, Roger Lipscombe > wrote: >> >> On 4 May 2018 at 12:07, Karolis Petrauskas > wrote: >> I agree with Heinz. The deprecation of gen_fsm forced me to stuck on OTP 19 and I dont see any reasonable path to upgrade. >> >> We upgraded successfully to OTP 20 (and compiled against OTP 21) by simply adding the following to the top of our gen_fsm modules: >> >> -compile([nowarn_deprecated_function]). % gen_fsm is deprecated; use gen_statem >> >> It'll be a problem when gen_fsm is *removed*, but until then, we'll stick with this. > > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From oliver.korpilla@REDACTED Fri May 4 14:29:57 2018 From: oliver.korpilla@REDACTED (Oliver Korpilla) Date: Fri, 04 May 2018 08:29:57 -0400 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> Message-ID: <929A5E12-CDF9-442D-8EC3-5267CBCD2E91@gmx.de> Ah yes, dependencies! I see your problem now. We're using gen_statem for a test framework written in elixir for telecoms. The FSM has been the least of our problems. I'm always amazed at the range of Erlang versions still in use. I keep forgetting that. Oliver On May 4, 2018 8:25:35 AM EDT, "Heinz N. Gies" wrote: >Hi Oliver, >I am not advocating to forbid migrating code to gen_statem when the >maintainer feels it is an improvement for their code. I?m advocating >for not forcing maintainers doing so for no reason. > >Karolis brought up another very good point I didn?t even think about. >Code written with gen_statem will not be able to run on older erlang >releases. While that?s probably not a issue on your own code where you >control the releases completely it?s a nightmare for open source and >libraries. > > >> On 4. May 2018, at 14:12, Oliver Korpilla >wrote: >> >> Except for the init bug in 19.0 (which our infrastructure was fixed >on for a long time) we found migrating to gen_statem a painless >experience with decent improvements. But maybe we simply use a >different or limited feature set? >> >> Oliver >> >> On May 4, 2018 8:03:11 AM EDT, "Heinz N. Gies" >wrote: >> I have thought of that but I don?t think that?s a real option. >Deprication warnings are important, this would turn all of them off >which doesn?t help code quality. I want to know if something is >deprecated and fix it in most cases. But as Karolis pointed out, it?s >easy for a function or two but for a whole behaviour it?s nearly >impossible. >> >> It basically creates two erlang worlds, pre-statem and post-statem. >> >> The stack trace is not great but it?s not as bad, it?s somewhat >possible to just wrap the whole function calling it into a ifdef and >call it a day. Not great it still is a decent amount of code >duplication but it?s possible. >https://github.com/Kyorai/clique/commit/b0e89de3cbb56396b59a1023903c7259ec2ef145#diff-7a19efbe2afedfb64aa7ec6690e9a13e > >would be an example. It?s more annoying then other changes but not >nearly as bad as the gen_fsm >> >>> On 4. May 2018, at 13:58, Roger Lipscombe > wrote: >>> >>> On 4 May 2018 at 12:07, Karolis Petrauskas > wrote: >>> I agree with Heinz. The deprecation of gen_fsm forced me to stuck on >OTP 19 and I dont see any reasonable path to upgrade. >>> >>> We upgraded successfully to OTP 20 (and compiled against OTP 21) by >simply adding the following to the top of our gen_fsm modules: >>> >>> -compile([nowarn_deprecated_function]). % gen_fsm is deprecated; >use gen_statem >>> >>> It'll be a problem when gen_fsm is *removed*, but until then, we'll >stick with this. >> >> >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From torben.lehoff@REDACTED Fri May 4 14:35:25 2018 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Fri, 04 May 2018 12:35:25 +0000 Subject: [erlang-questions] Why doesn't gen_statem allow {next_event, ...} for state enter? Message-ID: Hi, I have just been bitten by this: state1(enter, ...) -> ... {next_state, state2, NewData, {next_event, internal, just_do_it}}; but that is not a legal action for an enter thingy. Why is that so? Cheers, Torben -- https://www.linkedin.com/in/torbenhoffmann/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahesh@REDACTED Fri May 4 14:40:54 2018 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Fri, 4 May 2018 08:40:54 -0400 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: <929A5E12-CDF9-442D-8EC3-5267CBCD2E91@gmx.de> References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <929A5E12-CDF9-442D-8EC3-5267CBCD2E91@gmx.de> Message-ID: To echo Heinz's point, is there a *reason* for the deprecation? Enquiring minds want to know... On Fri, May 4, 2018 at 8:29 AM, Oliver Korpilla wrote: > Ah yes, dependencies! I see your problem now. > > We're using gen_statem for a test framework written in elixir for > telecoms. The FSM has been the least of our problems. > > I'm always amazed at the range of Erlang versions still in use. I keep > forgetting that. > > Oliver > > > On May 4, 2018 8:25:35 AM EDT, "Heinz N. Gies" wrote: >> >> Hi Oliver, >> I am not advocating to forbid migrating code to gen_statem when the >> maintainer feels it is an improvement for their code. I?m advocating for >> not forcing maintainers doing so for no reason. >> >> Karolis brought up another very good point I didn?t even think about. >> Code written with gen_statem will not be able to run on older erlang >> releases. While that?s probably not a issue on your own code where you >> control the releases completely it?s a nightmare for open source and >> libraries. >> >> >> On 4. May 2018, at 14:12, Oliver Korpilla wrote: >> >> Except for the init bug in 19.0 (which our infrastructure was fixed on >> for a long time) we found migrating to gen_statem a painless experience >> with decent improvements. But maybe we simply use a different or limited >> feature set? >> >> Oliver >> >> On May 4, 2018 8:03:11 AM EDT, "Heinz N. Gies" >> wrote: >>> >>> I have thought of that but I don?t think that?s a real option. >>> Deprication warnings are important, this would turn all of them off which >>> doesn?t help code quality. I want to know if something is deprecated and >>> fix it in most cases. But as Karolis pointed out, it?s easy for a function >>> or two but for a whole behaviour it?s nearly impossible. >>> >>> It basically creates two erlang worlds, pre-statem and post-statem. >>> >>> The stack trace is not great but it?s not as bad, it?s somewhat possible >>> to just wrap the whole function calling it into a ifdef and call it a day. >>> Not great it still is a decent amount of code duplication but it?s >>> possible. https://github.com/Kyorai/clique/commit/ >>> b0e89de3cbb56396b59a1023903c7259ec2ef145#diff- >>> 7a19efbe2afedfb64aa7ec6690e9a13e would be an example. It?s more >>> annoying then other changes but not nearly as bad as the gen_fsm >>> >>> On 4. May 2018, at 13:58, Roger Lipscombe >>> wrote: >>> >>> On 4 May 2018 at 12:07, Karolis Petrauskas >>> wrote: >>> >>>> I agree with Heinz. The deprecation of gen_fsm forced me to stuck on >>>> OTP 19 and I dont see any reasonable path to upgrade. >>>> >>> >>> We upgraded successfully to OTP 20 (and compiled against OTP 21) by >>> simply adding the following to the top of our gen_fsm modules: >>> >>> -compile([nowarn_deprecated_function]). % gen_fsm is deprecated; use >>> gen_statem >>> >>> It'll be a problem when gen_fsm is *removed*, but until then, we'll >>> stick with this. >>> >>> >>> >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >> >> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- *Mahesh Paolini-Subramanya That tall bald Indian guy..* *Twitter | Blog | G+ | LinkedIn * -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Fri May 4 14:40:51 2018 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Fri, 4 May 2018 14:40:51 +0200 Subject: [erlang-questions] [centos-7]With newer Erlang versions has the eliptic curve crypto situation changed? In-Reply-To: References: Message-ID: <681aabe1-092d-a56b-a518-c8f49da6741c@ericsson.com> The patch will be OTP-20.3.6 in next week. I'm sorry, but we will not do a patch on earlier 20 releases. However, the new crypto (crypto-4.2.2) *could* work in an OTP-20.2 environment, but I haven't tried. /Hans On 05/04/2018 02:07 PM, Nicholas Lundgaard wrote: > Hans, > > Regarding this issue, which 20.x versions will this patch be applied to? Is it possible to have a 20.2.x patch in addition to the one for the latest 20.3.x? > > If it's possible to know what the timeframe is on a patch release, I would really appreciate it. We deploy onto CentOS 6/7 at my company; just last night I was just working on updating our Erlang version from 20.1.7 to 20.2.4 and encountered this issue. We would remain on 20.1.7, but it has an issue with SSL Server Name Indication that have been fixed in 20.2[1] which has caused us problems recently. > > Thanks, > ?Nicholas Lundgaard > > [1]: https://github.com/erlang/otp/commit/78a9a09af9216a2dea454f561e0774e67a15c361 > >> From: Hans Nilsson R >> Subject: Re: [erlang-questions] [centos-7]With newer Erlang versions has the eliptic curve crypto situation changed? >> Date: May 4, 2018 at 4:54:29 AM CDT >> To: Bryan Hunt , >> >> >> Thanks! >> >> Will be fixed in OTP-21.0-rc2 and in a patch on OTP-20 which also has this error. >> >> It is the engine support down in openssl that has problems, so I simply disable our engine support if EC is disabled. >> >> /Hans >> >> On 05/03/2018 09:23 PM, Bryan Hunt wrote: >>> >>> Hi, >>> >>> Before, when building OTP I always set the compile time option to disable elliptic curve cryptography using the >>> CFLAGS environmental variable (RHEL doesn?t ship with it) : >>> >>> ``` >>> export CFLAGS="-DOPENSSL_NO_EC=1" >>> ./otp_build configure \ >>> --without-odbc \ >>> --without-cosEventDomain \ >>> --without-cosEvent \ >>> --without-cosFileTransfer \ >>> --without-cosNotification \ >>> --without-cosProperty \ >>> --without-cosTime \ >>> --without-cosTransactions \ >>> --without-debugger \ >>> --without-et \ >>> --without-gs \ >>> --without-ic \ >>> --without-javac \ >>> --without-jinterface \ >>> --without-megaco \ >>> --without-observer \ >>> --without-orber \ >>> --without-percept \ >>> --without-typer \ >>> --without-wx \ >>> --without-tv \ >>> --without-diameter \ >>> --without-hipe >>> ``` >>> >>> And that still works for the older versions. >>> >>> But when applied to OTP-21.0-rc1 I receive the following error : >>> >>> ``` >>> gmake[6]: Entering directory `/root/otp/lib/crypto/c_src' >>> CC ../priv/obj/x86_64-unknown-linux-gnu/crypto.o >>> In file included from /usr/include/openssl/ecdh.h:78:0, >>> from /usr/include/openssl/engine.h:86, >>> from crypto.c:63: >>> /usr/include/openssl/ec.h:82:4: error: #error EC is disabled. >>> # error EC is disabled. >>> ^ >>> gmake[6]: Leaving directory `/root/otp/lib/crypto/c_src' >>> gmake[6]: *** [../priv/obj/x86_64-unknown-linux-gnu/crypto.o] Error 1 >>> gmake[5]: *** [release_spec] Error 2 >>> gmake[5]: Leaving directory `/root/otp/lib/crypto/c_src' >>> gmake[4]: *** [release] Error 2 >>> gmake[4]: Leaving directory `/root/otp/lib/crypto/c_src' >>> gmake[3]: *** [release] Error 2 >>> gmake[3]: Leaving directory `/root/otp/lib/crypto/c_src' >>> gmake[2]: *** [release] Error 2 >>> gmake[2]: Leaving directory `/root/otp/lib/crypto' >>> gmake[1]: Leaving directory `/root/otp/lib' >>> gmake[1]: *** [release] Error 2 >>> gmake: *** [release] Error 2 >>> The command '/bin/sh -c ./build-erlang.sh' returned a non-zero code: 1 >>> Unable to find image 'bryanhuntesl/centos7-erlang:OTP-21.0-rc1' locally >>> docker: Error response from daemon: manifest for bryanhuntesl/centos7-erlang:OTP-21.0-rc1 not found. >>> See 'docker run --help?. >>> ``` >>> >>> Has this behaviour changed recently ? >>> >>> Bryan >>> >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> > > From essen@REDACTED Fri May 4 14:57:18 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Fri, 4 May 2018 14:57:18 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> Message-ID: <791172d4-5d4f-2f85-b9fd-644885accea7@ninenines.eu> On 05/04/2018 02:03 PM, Heinz N. Gies wrote: > I have thought of that but I don?t think that?s a real option. > Deprication warnings are important Are they? Code removal is an important problem which can't be avoided when you have to support OTP versions before/after. So is fundamental additions to the language (like maps). But I don't think warnings are nearly as important, particularly deprecation warnings. Deprecation warnings signal that your code will stop working one day in the future so they are useful, but they say nothing of your code TODAY which is what matters most. The code still builds and works fine regardless of the warning. I would say it's important to know that the code will stop working at some point in the future so you can plan migrating the code. But you still have at least a year to figure it out, it's not OTP-21 that breaks your code, it will be OTP-22 or a later release. If I was in your situation I would adopt the following strategy: * Disable the deprecation warnings in normal builds * Setup a separate build task/CI job for compiling with deprecation warnings (xref checks those too IIRC) * Open a ticket to remember to fix those and/or change the supported OTP versions within that timespan Dropping older versions will work even for many libraries because most users are on the last 2-3 releases anyway. There's of course exceptions. In general I would also say that while it's usually a good idea to fix new warnings, sometimes the fix is uglier than the warning. If you need to depend on externally defined macros or rewrite huge chunks of the code to fix a warning, you might want to just leave the warning around or disable it for now. Warnings are great to anticipate problems but they should by no means prevent working code from working. All that being said, I think the OTP team has been a little too aggressive with changes lately. It would be better to deprecate two releases before removal (introduce gen_statem in 20, deprecate gen_fsm in 21 and remove gen_fsm in 23). That would make transitioning easier because one could switch from gen_fsm to gen_statem during the OTP-22 era while still supporting the last 3 releases and therefore the majority of users would be unaffected. Cheers, -- Lo?c Hoguin https://ninenines.eu From raimo+erlang-questions@REDACTED Fri May 4 15:30:25 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 4 May 2018 15:30:25 +0200 Subject: [erlang-questions] Why doesn't gen_statem allow {next_event, ...} for state enter? In-Reply-To: References: Message-ID: <20180504133025.GA4698@erix.ericsson.se> On Fri, May 04, 2018 at 12:35:25PM +0000, Torben Hoffmann wrote: > Hi, > > I have just been bitten by this: > state1(enter, ...) -> > ... > {next_state, state2, NewData, > {next_event, internal, just_do_it}}; > > but that is not a legal action for an enter thingy. > > Why is that so? That bug was fixed in OTP-20.3.2. > > Cheers, > Torben > -- > https://www.linkedin.com/in/torbenhoffmann/ -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From mononcqc@REDACTED Fri May 4 15:31:07 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 4 May 2018 09:31:07 -0400 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: <791172d4-5d4f-2f85-b9fd-644885accea7@ninenines.eu> References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <791172d4-5d4f-2f85-b9fd-644885accea7@ninenines.eu> Message-ID: Overall, I like some of the aggressiveness that we have seen lately on introducing new useful features and seeing older ones get removed. I hate that I'll have to adapt a bunch of projects to changes in logger interfaces, but it will definitely be a more positive experience for users joining the platform now. Similarly for Unicode support in the strings module. The thing that I'm a bit unsure about is that deprecation is used as a warning to say something is: a) no longer the favored mechanism to do something b) will eventually be removed. I can tolerate warnings for a) and ignore them, but I don't want to do it for b) unless I do not know how long a thing will be deprecated for. The thing is that if the warnings are used for a), you can have it deprecated for 10 years and it won't be a big deal; gen_fsm and the old string interface *might* fall into that category. Maybe get_stacktrace() as well? But for the case of b), if I don't have a straight timeline of when a feature is going to disappear, then you bet your ass I want warnings_as_errors and to tackle it ASAP. Because if I'm working on a big project, there's no telling if a migration will take me 2 weeks or 2 years (see how long Riak has been stuck on old ass versions), and the more lenient you are as an organisation, the harder it is to stop being so. I think I have something like 30-40 open-source libraries right now, most of which are considered 'done' and no longer need changes. When the string handling landed, I had to update dozens of them, and also had to shepherd dozens of other ones from the community at large, sometimes because maintainers have left, or sometimes because their upgrade schedule does not align with mine. If I know a deprecation means "something newer is being used but we're not thinking of removing the old one" then fine. But right now I don't know which it is I am getting, and that is where things get a bit frustrating. I have to assume "Now is better than later": I don't want to be left out of a critical SSL fix because I have a few thousand lines of non-critical code that need patching up to get up there, delaying a deploy or a release by a few precious days. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Fri May 4 15:40:24 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 4 May 2018 15:40:24 +0200 Subject: [erlang-questions] gen_statem is blocking in gen do_call, a lot of messages does not be handled. In-Reply-To: References: Message-ID: <20180504134024.GB4698@erix.ericsson.se> On Fri, May 04, 2018 at 03:51:45PM +0800, getonga wrote: > The process info is > ``` > process_info(pid(0, 818, 0)). > [{current_function,{gen,do_call,4}}, > {initial_call,{proc_lib,init_p,5}}, > {status,waiting}, > {message_queue_len,10}, > {messages,[1,2,3,3,4,5,6,7,8,9,10]}, > {links,[<0.514.0>,<0.1294.0>]}, > {dictionary,[{rand_seed,{#{bits => 58,jump => #Fun, > next => #Fun,type => exrop, > uniform => #Fun, > uniform_n => #Fun,weak_low_bits => 1}, > [218965415455180470|167242355698640157]}}, > {'$initial_call',{time_server,init,1}}, > {'$ancestors',[time_sup,app_sup,<0.512.0>]}]}, > {trap_exit,true}, > {error_handler,error_handler}, > {priority,normal}, > {group_leader,<0.511.0>}, > {total_heap_size,987}, > {heap_size,987}, > {stack_size,47}, > {reductions,27674}, > {garbage_collection,[{max_heap_size,#{error_logger => true,kill => true,size => 0}}, > {min_bin_vheap_size,46422}, > {min_heap_size,233}, > {fullsweep_after,65535}, > {minor_gcs,0}]}, > {suspending,[]}] > > ``` If you have a gen_statem server that from its event handler calls some other server via gen:do_call, e.g from gen_server:call, and that call hangs there is nothing the calling server can do about it. Do not make blocking calls from an event handler! That will prevent the server from processing it's message queue. Find out why the called server is blocked. You may have two servers calling each other which is a way to create deadlock... > > > and I look into the gen_statem.erl code, the gen:call/4 is running without timeout: > ``` > call_dirty(ServerRef, Request, Timeout, T) -> > try gen:call(ServerRef, '$gen_call', Request, T) of > {ok,Reply} -> > Reply > catch > Class:Reason -> > erlang:raise( > Class, > {Reason,{?MODULE,call,[ServerRef,Request,Timeout]}}, > erlang:get_stacktrace()) > end. > > ``` I do not know where you are getting with this, but the argument T contains the timeout time. The argument Timeout is only used when raising an error to fake (with the right arguments) that it was gen_statem:call/3 that caused the exception. > > > Is the code blocks my gen_statem process ? How can I disable it? Do not do blocking calls from an event handler! -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From vladdu55@REDACTED Fri May 4 15:52:50 2018 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Fri, 04 May 2018 13:52:50 +0000 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <791172d4-5d4f-2f85-b9fd-644885accea7@ninenines.eu> Message-ID: On Fri, May 4, 2018 at 3:31 PM Fred Hebert wrote: > The thing that I'm a bit unsure about is that deprecation is used as a > warning to say something is: > > a) no longer the favored mechanism to do something > b) will eventually be removed. > > I can tolerate warnings for a) and ignore them, but I don't want to do it > for b) unless I do not know how long a thing will be deprecated for. > Maybe there could be two different warning for the two use cases? I confess I can't come up with a good word for the other, though... regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliver.korpilla@REDACTED Fri May 4 16:17:28 2018 From: oliver.korpilla@REDACTED (Oliver Korpilla) Date: Fri, 04 May 2018 10:17:28 -0400 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <791172d4-5d4f-2f85-b9fd-644885accea7@ninenines.eu> Message-ID: "Being phased out by"? Oliver On May 4, 2018 9:52:50 AM EDT, Vlad Dumitrescu wrote: >On Fri, May 4, 2018 at 3:31 PM Fred Hebert wrote: > >> The thing that I'm a bit unsure about is that deprecation is used as >a >> warning to say something is: >> >> a) no longer the favored mechanism to do something >> b) will eventually be removed. >> >> I can tolerate warnings for a) and ignore them, but I don't want to >do it >> for b) unless I do not know how long a thing will be deprecated for. >> > >Maybe there could be two different warning for the two use cases? I >confess >I can't come up with a good word for the other, though... > >regards, >Vlad -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott_ribe@REDACTED Fri May 4 16:21:12 2018 From: scott_ribe@REDACTED (Scott Ribe) Date: Fri, 4 May 2018 08:21:12 -0600 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <791172d4-5d4f-2f85-b9fd-644885accea7@ninenines.eu> Message-ID: <39C6B6F1-97E3-44C0-9C14-732A9C27E940@elevated-dev.com> "endangered" vs "deprecated"? ;-) -- Scott Ribe scott_ribe@REDACTED https://www.linkedin.com/in/scottribe/ > On May 4, 2018, at 8:17 AM, Oliver Korpilla wrote: > > "Being phased out by"? > > Oliver > > On May 4, 2018 9:52:50 AM EDT, Vlad Dumitrescu wrote: > > > On Fri, May 4, 2018 at 3:31 PM Fred Hebert wrote: > The thing that I'm a bit unsure about is that deprecation is used as a warning to say something is: > > a) no longer the favored mechanism to do something > b) will eventually be removed. > > I can tolerate warnings for a) and ignore them, but I don't want to do it for b) unless I do not know how long a thing will be deprecated for. > > Maybe there could be two different warning for the two use cases? I confess I can't come up with a good word for the other, though... > > regards, > Vlad > > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From vladdu55@REDACTED Fri May 4 16:31:23 2018 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Fri, 04 May 2018 14:31:23 +0000 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: <39C6B6F1-97E3-44C0-9C14-732A9C27E940@elevated-dev.com> References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <791172d4-5d4f-2f85-b9fd-644885accea7@ninenines.eu> <39C6B6F1-97E3-44C0-9C14-732A9C27E940@elevated-dev.com> Message-ID: Maybe "discouraged"? /Vlad On Fri, May 4, 2018 at 4:21 PM Scott Ribe wrote: > "endangered" vs "deprecated"? > > ;-) > > -- > Scott Ribe > scott_ribe@REDACTED > https://www.linkedin.com/in/scottribe/ > > > > > On May 4, 2018, at 8:17 AM, Oliver Korpilla > wrote: > > > > "Being phased out by"? > > > > Oliver > > > > On May 4, 2018 9:52:50 AM EDT, Vlad Dumitrescu > wrote: > > > > > > On Fri, May 4, 2018 at 3:31 PM Fred Hebert wrote: > > The thing that I'm a bit unsure about is that deprecation is used as a > warning to say something is: > > > > a) no longer the favored mechanism to do something > > b) will eventually be removed. > > > > I can tolerate warnings for a) and ignore them, but I don't want to do > it for b) unless I do not know how long a thing will be deprecated for. > > > > Maybe there could be two different warning for the two use cases? I > confess I can't come up with a good word for the other, though... > > > > regards, > > Vlad > > > > > > -- > > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From heinz@REDACTED Fri May 4 16:34:35 2018 From: heinz@REDACTED (Heinz N. Gies) Date: Fri, 4 May 2018 16:34:35 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <791172d4-5d4f-2f85-b9fd-644885accea7@ninenines.eu> <39C6B6F1-97E3-44C0-9C14-732A9C27E940@elevated-dev.com> Message-ID: <9E3D4EA8-0260-48E1-B49D-9043016C3262@licenser.net> I really like discouraged. It emphasises that you probably shouldn?t do it any more but that OTP won?t blow up your program next year if you do. ?? and it?s shorter then ?Please don?t do that any more but if you did before we?re not mad? ;) > On 4. May 2018, at 16:31, Vlad Dumitrescu wrote: > > Maybe "discouraged"? > /Vlad > > > On Fri, May 4, 2018 at 4:21 PM Scott Ribe > wrote: > "endangered" vs "deprecated"? > > ;-) > > -- > Scott Ribe > scott_ribe@REDACTED > https://www.linkedin.com/in/scottribe/ > > > > > On May 4, 2018, at 8:17 AM, Oliver Korpilla > wrote: > > > > "Being phased out by"? > > > > Oliver > > > > On May 4, 2018 9:52:50 AM EDT, Vlad Dumitrescu > wrote: > > > > > > On Fri, May 4, 2018 at 3:31 PM Fred Hebert > wrote: > > The thing that I'm a bit unsure about is that deprecation is used as a warning to say something is: > > > > a) no longer the favored mechanism to do something > > b) will eventually be removed. > > > > I can tolerate warnings for a) and ignore them, but I don't want to do it for b) unless I do not know how long a thing will be deprecated for. > > > > Maybe there could be two different warning for the two use cases? I confess I can't come up with a good word for the other, though... > > > > regards, > > Vlad > > > > > > -- > > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From lloyd@REDACTED Fri May 4 16:39:34 2018 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Fri, 4 May 2018 10:39:34 -0400 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: Hi Heinz, This exactly how I felt when I found that string functions I relied on had been dropped. Perhaps we need a library called ?orphans.? Best wishes, Lloyd Sent from my iPad > On May 4, 2018, at 6:18 AM, Heinz N. Gies wrote: > > Since this might get long I want to start with an anecdote. > > When talking to people about Erlang and why it is great one of the things I mention is stability. I?m not talking about systems not crashing but rather about the fact that change are slow, gentle and breaking changes are rare or nearly unheard off. The example I use is a non trivial toy program I wrote back in 2012 and that it took me a whole of 20 minutes to get code form 2012 working 4 and a half years later (most of the time spent on updating dependencies). > > Having used other languages and the constant catch up game with changes I always found that impressive and it gives me a warm and fuzzy feeling inside that the code I finish will be finished and don?t require a endless cat and mouse with the language. > > Even changes that happen like random vs rand are well based with reason - security and performance in that case, and the changes are minor enough that for the most case changing a single function call here and there is enough to even keep complex programs as long as they?re not (ab)using internal knowledge of the implementation (I?m looking at you riak_core). > > This stability has always be a core value of the Erlang eco system to me, and an important one. > > While I have the feeling the frequency of deprivations and breaking cross version compatibility has increased over the last releases for most of them I can, as said before, see the reason and the tradeoff made. > > Now lets talk about senseless murder of gen_fsm. > > I understand that gen_statem is nicer, and more user friendly, and I don?t want to advocate against it at all, it?s great that we have it and I see forward to useing it for the next state machine I have to write! > > The next one. I see forward using it for the next one, not the last 100, or even last 15 not even for the last two. I don?t see forward to being forced to go through all libraries that have perfectly fine working gen_fsm without problems or troubles and have to change them. > > Ripping out a widely used and not broken behaviour just because there is a more fancy one goes against everything I value in erlang as an eco system. It forces developers to do the kind of busy work I detest in other platforms. > > Worst of all forcing people to fundamentally re-write perfectly good code means bugs, bugs we wouldn?t have had if we would just have left the code alone. Bugs that are probably non obvious as differences are subtile. And in the end for absolutely no benefit. > > Now it?s just a rant without a proposal. And I detest whining without constructive criticism as much as I detest platforms forcing changes that are not required. So here is what I?d like to suggest: > > - Keep gem_fsm around, it has been a friend to many for a long time and there is a lot of code using it that shouldn?t have to change to use newer erlang versions. > - Encourage people to use gen_statm for new code, and if it is really that much better then gen_fsm that it?d warrant to rip it out that should happen on it?s own. > > Heinz - member of team gen_fsm ?? > > PS: I?m not just writing this out of no-where I?ve been fighting subtile bugs on gen_fsm -> gem_statem code that was ported by people a lot smarter then me and I see no chance in hell that if they can?t pull that off on simple code I will ever be able to do that on complex code. > > PPS: if the this plea falls on deaf ears (which I really hope it does?t) here is a plan-b for team gen_fsm https://gitlab.com/Project-FiFo/gen_fsm_compat > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlangsiri@REDACTED Fri May 4 16:45:40 2018 From: erlangsiri@REDACTED (Siri Hansen) Date: Fri, 4 May 2018 16:45:40 +0200 Subject: [erlang-questions] Fwd: Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <6923ce2d46042004b9dbf153a77a79ff73e30185.camel@applicata.bg> Message-ID: Sorry - forgot to copy list... ---------- Forwarded message ---------- From: Siri Hansen Date: 2018-05-04 16:44 GMT+02:00 Subject: Re: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) To: Svilen Ivanov Hi, thanks for reporting - it should be removing_handler/2 - I will correct it asap. Regards /siri 2018-05-03 12:48 GMT+02:00 Svilen Ivanov : > In Kernel User's Guide new logger API chapter > "2.6 Example: implement a handler" > callback function 'removing_handler' is documented to accept single > parameter logger:handler_id(), > but in the example for handler that prints to file it takes two > parameters (Id and Config). > > The current implementation seems to call this function with single > parameter HandlerId, but that makes not possible to cleanly close the > file descriptor as shown in the example. > > So, removing_handler/1 or removing_handler/2 ? > > BR, Svilen > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.schultz@REDACTED Fri May 4 16:58:51 2018 From: andreas.schultz@REDACTED (Andreas Schultz) Date: Fri, 04 May 2018 14:58:51 +0000 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <1525261079.4069.73.camel@ericsson.com> References: <1525261079.4069.73.camel@ericsson.com> Message-ID: Hi, The RC breaks rebar3 badly, not sure if it's rebar's fault or a problem with the new logger... With an up to date rebar3 clone: rebar3$ ./bootstrap (no logger present) unexpected logger message: {log,error,"Error in process ~p with exit value:~n~p~n",[<0.39.0>,{badarg,[{ets,lookup,[logger,proc_lib],[]},{logger_config,allow,3,[{file,"logger_config.erl"},{line,42}]},{proc_lib,crash_report,4,[{file,"proc_lib.erl"},{line,508}]},{proc_lib,exit_p,3,[{file,"proc_lib.erl"},{line,269}]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.37.0>,pid=><0.39.0>,time=>-576460751953591}} (no logger present) unexpected logger message: {log,error,"Error in process ~p with exit value:~n~p~n",[<0.39.0>,{badarg,[{ets,lookup,[logger,proc_lib],[]},{logger_config,allow,3,[{file,"logger_config.erl"},{line,42}]},{proc_lib,crash_report,4,[{file,"proc_lib.erl"},{line,508}]},{proc_lib,exit_p,3,[{file,"proc_lib.erl"},{line,269}]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.37.0>,pid=><0.39.0>,time=>-576460751953591}} ... And so on... Regards Andreas Henrik Nord X schrieb am Mi., 2. Mai 2018 um 13:38 Uhr: > OTP 21 Release Candidate 1 > > This is the first of two planned release candidates before the OTP 21 > release. The intention whit this release is that you as users try it > and give us feedback if something does not work as expected. Could be a > bug,an unexpected incompatibility, a significant change of > characteristics in a negative direction, etc. > > Erlang/OTP 21 is a new major release with new features, improvements as > well as incompatibilities. > > Potential Incompatibilities > > * All Corba applications are now moved from the OTP repository > * A new Corba repository will be created https://github.com/erlang > * New applications ftp and tftp, moved from inets > * ssl no longer supports 3_DES cipher suites or RSA-key exchange > cipher suites by default > * erlang:monitor on a primitive node (erl_interface, jinterface, etc) > will no longer fail with badarg exception. Instead a monitor will be > created, but it will only supervise the connection to the node. > > Highlights > > Erts: > > * Enhanced IO scalability > * Support for usage of distribution controller processes for > alternative transports, routing etc > * compact instructions on 64bit systems for code below 4GB 20% less > memory for loaded code > * Rewrite of the efile-driver with NIFs and "Dirty schedulers" > resulting in faster file operations > * non-smp VM removed > * link and monitor optimized for scalability > * os:getenv/putenv now work on thread-safe emulation. No longer in > sync with libc getenv(3). Manual synchronization will be needed. > > Compiler: > * Misc compiler optimizations including contributions from the > Elixir team resulting in 10% improvements in benchmarks > * "Tuple calls" have been removed from the run-time system. > * Code such as f({ok, Val}) -> {ok, Val} is now automatically > rewritten to f({ok, Val} = Tuple) -> Tuple. this reduces code > size, execution time, and removed GC pressure. > * More information in stacktrace from a number of operators > * erlang:get_stacktrace/0 deprecated to be replaced with try ... > catch C:R:Stacktrace -> ... > * Creation of small maps with literal keys optimized. > > Security: > * DTLS support in SSL > * Enhanced support for distribution over TLS > * "unsecure" ciphers removed from defaults in SSL and SSH. > * A new option value defined to facilitate implementing exec > servers. Old option kept for compatibility, but now gives errors > on stderror. > > Standard libraries: > * New API for logging, logger > * New uri_string module for parsing URIs according to "The > standard" > * New function lists:search(list,fun/1) -> {ok, Value} | false > * Changed default behaviour of .erlang loading. escript, erlc, > dialyzer and typer no longer load an .erlang at all. > > For more details see > http://erlang.org/download/otp_src_21.0-rc1.readme > > Pre built versions for Windows can be fetched here: > http://erlang.org/download/otp_win32_21.0-rc1.exe > http://erlang.org/download/otp_win64_21.0-rc1.exe > > Online documentation can be browsed here: > http://erlang.org/documentation/doc-10.0-rc1/doc/ > > The Erlang/OTP source can also be found at GitHub on the official > Erlang repository, > https://github.com/erlang/otp with tag OTP-21.0-rc1 > > > Thank you for all your contributions! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- -- Dipl.-Inform. Andreas Schultz ----------------------- enabling your networks ---------------------- Travelping GmbH Phone: +49-391-81 90 99 0 Roentgenstr. 13 Fax: +49-391-81 90 99 299 39108 Magdeburg Email: info@REDACTED GERMANY Web: http://www.travelping.com Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 --------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri May 4 17:41:41 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 4 May 2018 17:41:41 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Message-ID: Hi! Thank you for the example I did find one bug, the patch is here: diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl index 0956d35..dd3dc54 100644 --- a/lib/ssl/src/ssl_cipher.erl +++ b/lib/ssl/src/ssl_cipher.erl @@ -2837,7 +2837,7 @@ key_uses(OtpCert) -> Extensions = ssl_certificate:extensions_list(TBSExtensions), case ssl_certificate:select_extension(?'id-ce-keyUsage', Extensions) of undefined -> - undefined; + []; #'Extension'{extnValue = KeyUses} -> KeyUses end. my other sslv2 issue seems not to be related. (Probably a OpenSSL issue) Your example does however not work perfect so I am continuing to look into this! Regards Ingela Erlang/OTP team - Ericsson AB 2018-05-04 12:25 GMT+02:00 Roger Lipscombe : > On 4 May 2018 at 08:32, Ingela Andin wrote: > > This error is consistent with one of the errors I am seeing in the > nightly > > builds when running OpenSSL with only default parameters so I suspect > > something is off in combination > > version negotiation and cipher suite selection checks. I am looking in to > > it! > > I'm seeing the same, if it helps to reproduce. I generated my certificates > with: > > #!/bin/sh > > # Create the CA key and certificate. > openssl genrsa -out ca.key 2048 > openssl req -new -x509 -nodes -key ca.key -days 3653 -out ca.pem -subj > "/CN=Test CA" > > # Create the server key and CSR. > openssl genrsa -out server.key 2048 > openssl req -new -key server.key -out server.csr -subj "/CN=localhost" > > # Sign the CSR with the CA key. > serial=$(date +"%s") > openssl x509 -req -days 3563 -CA ca.pem -CAserial $serial > -CAcreateserial -CAkey ca.key -in server.csr -out server.pem > rm $serial > > I tested with: > > % Server > {ok, _} = application:ensure_all_started(ssl). > Port = 11002. > LOpts = [{certfile, "server.pem"}, {keyfile, "server.key"}]. > {ok, LSock} = ssl:listen(Port, LOpts). > {ok, CSock} = ssl:transport_accept(LSock). > ok = ssl:ssl_accept(CSock). > > % Client > {ok, _} = application:ensure_all_started(ssl). > Port = 11002. > COpts = [{verify, verify_peer}, {cacertfile, "ca.pem"}]. > {ok, Sock} = ssl:connect("localhost", Port, COpts). > > The server reports: > > =INFO REPORT==== 4-May-2018::11:22:20.971130 === > TLS server: In state hello at tls_handshake.erl:130 generated SERVER > ALERT: Fatal - Handshake Failure - malformed_handshake_data > > ** exception error: no match of right hand side value > {error,{tls_alert,"handshake failure"}} > > The client reports: > > =INFO REPORT==== 4-May-2018::11:22:20.981524 === > TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure > > ** exception error: no match of right hand side value > {error,{tls_alert,"handshake failure"}} > > The same code works fine with 20.3.1 > > Thanks, > Roger. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Fri May 4 18:19:59 2018 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Fri, 04 May 2018 16:19:59 +0000 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: Golang has something that solves this ?busy work? problem by writing one off programs that manipulate the AST then pretty print your code, updated: gofix. C/Linux has something which I think is more powerful: Coccinelle & semantic patches. Facebook has a tool called pfff (not sure how many Fs) that implements that ?spatch? language and I believe supports Erlang (though the primary usage is targeted at PHP). As I have said on this list before I would gladly help writing a linter for Erlang (hopefully for Core Erlang) that would provide code suggestions. Well, I?d gladly work on an Spatch for Erlang and help provide for example an Spatch file to automatically go from a gen_fsm to gen_statem implementation. Surely this has rough edges! But life is short and full of terrors. On Fri 4 May 2018 at 12:18, Heinz N. Gies wrote: > Since this might get long I want to start with an anecdote. > > When talking to people about Erlang and why it is great one of the things > I mention is stability. I?m not talking about systems not crashing but > rather about the fact that change are slow, gentle and breaking changes are > rare or nearly unheard off. The example I use is a non trivial toy program > I wrote back in 2012 and that it took me a whole of 20 minutes to get code > form 2012 working 4 and a half years later (most of the time spent on > updating dependencies). > > Having used other languages and the constant catch up game with changes I > always found that impressive and it gives me a warm and fuzzy feeling > inside that the code I finish will be finished and don?t require a endless > cat and mouse with the language. > > Even changes that happen like random vs rand are well based with reason - > security and performance in that case, and the changes are minor enough > that for the most case changing a single function call here and there is > enough to even keep complex programs as long as they?re not (ab)using > internal knowledge of the implementation (I?m looking at you riak_core). > > This stability has always be a core value of the Erlang eco system to me, > and an important one. > > While I have the feeling the frequency of deprivations and breaking cross > version compatibility has increased over the last releases for most of them > I can, as said before, see the reason and the tradeoff made. > > Now lets talk about senseless murder of gen_fsm. > > I understand that gen_statem is nicer, and more user friendly, and I don?t > want to advocate against it at all, it?s great that we have it and I see > forward to useing it for the next state machine I have to write! > > The *next* one. I see forward using it *for the next one*, not the last > 100, or even last 15 not even for the last two. I don?t see forward to > being forced to go through all libraries that have perfectly fine working > gen_fsm without problems or troubles and have to change them. > > Ripping out a widely used and not broken behaviour just because there is a > more fancy one goes against everything I value in erlang as an eco system. > It forces developers to do the kind of busy work I detest in other > platforms. > > Worst of all forcing people to fundamentally re-write perfectly good code > means bugs, bugs we wouldn?t have had if we would just have left the code > alone. Bugs that are probably non obvious as differences are subtile. And > in the end for absolutely no benefit. > > Now it?s just a rant without a proposal. And I detest whining without > constructive criticism as much as I detest platforms forcing changes that > are not required. So here is what I?d like to suggest: > > - Keep gem_fsm around, it has been a friend to many for a long time and > there is a lot of code using it that shouldn?t have to change to use newer > erlang versions. > - Encourage people to use gen_statm for new code, and if it is really that > much better then gen_fsm that it?d warrant to rip it out that should happen > on it?s own. > > Heinz - member of team gen_fsm ?? > > PS: I?m not just writing this out of no-where I?ve been fighting subtile > bugs on gen_fsm -> gem_statem code that was ported by people a lot smarter > then me and I see no chance in hell that if they can?t pull that off on > simple code I will ever be able to do that on complex code. > > PPS: if the this plea falls on deaf ears (which I really hope it does?t) > here is a plan-b for team gen_fsm > https://gitlab.com/Project-FiFo/gen_fsm_compat > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Cheers, -- Pierre Fenoll -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Fri May 4 18:23:23 2018 From: askjuise@REDACTED (Alexander Petrovsky) Date: Fri, 4 May 2018 19:23:23 +0300 Subject: [erlang-questions] gen_server and exit(Pid, shutdown) Message-ID: Hello! After a long timer working with Erlang and using gen_server abstraction, I've discovered for myself that when I send exit(Pid, shutdown) for some gen_server (or when application stopping), according to the documentation, terminate callback will be called only when trap_exit flag is true. In my mind it's absolutely not intuitively, the documentation very clear for trap_exit and it's says: When a process is trapping exits, it does not terminate when an exit signal > is received. Instead, the signal is transformed into a message > {'EXIT',FromPid,Reason}, which is put into the mailbox of the process, > just like a regular message. So, I suppose to see the message {'EXIT', FromPid, shutdown} in handle_info when I call exit(GenServerPid, shutdown), and I always suppose, that terminate callback will be called (of course I remeber about reason kill) -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 931 9877991 -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Fri May 4 18:55:43 2018 From: t@REDACTED (Tristan Sloughter) Date: Fri, 04 May 2018 09:55:43 -0700 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> Message-ID: <1525452943.3653629.1361100560.1798DB07@webmail.messagingengine.com> We cut a new release today, 3.5.2, that builds with 21 fine. Tristan On Fri, May 4, 2018, at 7:58 AM, Andreas Schultz wrote: > Hi, > > The RC breaks rebar3 badly, not sure if it's rebar's fault or a > problem with the new logger...> > With an up to date rebar3 clone: > > rebar3$ ./bootstrap > (no logger present) unexpected logger message: {log,error,"Error > in process ~p with exit value:~n~p~n",[<0.39.0>,{badarg,[{ets,loo- > kup,[logger,proc_lib],[]},{logger_config,allow,3,[{file,"logger_c- > onfig.erl"},{line,42}]},{proc_lib,crash_report,4,[{file,"proc_lib- > .erl"},{line,508}]},{proc_lib,exit_p,3,[{file,"proc_lib.erl"},{li- > ne,269}]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.- > 37.0>,pid=><0.39.0>,time=>-576460751953591}}> (no logger present) unexpected logger message: {log,error,"Error > in process ~p with exit value:~n~p~n",[<0.39.0>,{badarg,[{ets,loo- > kup,[logger,proc_lib],[]},{logger_config,allow,3,[{file,"logger_c- > onfig.erl"},{line,42}]},{proc_lib,crash_report,4,[{file,"proc_lib- > .erl"},{line,508}]},{proc_lib,exit_p,3,[{file,"proc_lib.erl"},{li- > ne,269}]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.- > 37.0>,pid=><0.39.0>,time=>-576460751953591}}> ... > > And so on... > > Regards > Andreas > > Henrik Nord X schrieb am Mi., 2. Mai 2018 > um 13:38 Uhr:>> OTP 21 Release Candidate 1 >> >> This is the first of two planned release candidates before the >> OTP 21>> release. The intention whit this release is that you as users try it>> and give us feedback if something does not work as expected. >> Could be a>> bug,an unexpected incompatibility, a significant change of >> characteristics in a negative direction, etc. >> >> Erlang/OTP 21 is a new major release with new features, >> improvements as>> well as incompatibilities. >> >> Potential Incompatibilities >> >> * All Corba applications are now moved from the OTP repository >> * A new Corba repository will be created >> https://github.com/erlang>> * New applications ftp and tftp, moved from inets >> * ssl no longer supports 3_DES cipher suites or RSA-key exchange >> cipher suites by default >> * erlang:monitor on a primitive node (erl_interface, jinterface, >> etc)>> will no longer fail with badarg exception. Instead a monitor will be>> created, but it will only supervise the connection to the node. >> >> Highlights >> >> Erts: >> >> * Enhanced IO scalability >> * Support for usage of distribution controller processes for >> alternative transports, routing etc >> * compact instructions on 64bit systems for code below 4GB 20% >> less>> memory for loaded code >> * Rewrite of the efile-driver with NIFs and "Dirty schedulers" >> resulting in faster file operations >> * non-smp VM removed >> * link and monitor optimized for scalability >> * os:getenv/putenv now work on thread-safe emulation. No longer >> in>> sync with libc getenv(3). Manual synchronization will be >> needed.>> >> Compiler: >> * Misc compiler optimizations including contributions from the >> Elixir team resulting in 10% improvements in benchmarks >> * "Tuple calls" have been removed from the run-time system. >> * Code such as f({ok, Val}) -> {ok, Val} is now automatically >> rewritten to f({ok, Val} = Tuple) -> Tuple. this reduces code>> size, execution time, and removed GC pressure. >> * More information in stacktrace from a number of operators >> * erlang:get_stacktrace/0 deprecated to be replaced with try ...>> catch C:R:Stacktrace -> ... >> * Creation of small maps with literal keys optimized. >> >> Security: >> * DTLS support in SSL >> * Enhanced support for distribution over TLS >> * "unsecure" ciphers removed from defaults in SSL and SSH. >> * A new option value defined to facilitate implementing exec >> servers. Old option kept for compatibility, but now gives >> errors>> on stderror. >> >> Standard libraries: >> * New API for logging, logger >> * New uri_string module for parsing URIs according to "The >> standard" >> * New function lists:search(list,fun/1) -> {ok, Value} | false >> * Changed default behaviour of .erlang loading. escript, erlc, >> dialyzer and typer no longer load an .erlang at all. >> >> For more details see >> http://erlang.org/download/otp_src_21.0-rc1.readme >> >> Pre built versions for Windows can be fetched here: >> http://erlang.org/download/otp_win32_21.0-rc1.exe >> http://erlang.org/download/otp_win64_21.0-rc1.exe >> >> Online documentation can be browsed here: >> http://erlang.org/documentation/doc-10.0-rc1/doc/ >> >> The Erlang/OTP source can also be found at GitHub on the official >> Erlang repository, >> https://github.com/erlang/otp with tag OTP-21.0-rc1 >> >> >> Thank you for all your contributions! >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -- > -- > Dipl.-Inform. Andreas Schultz > > ----------------------- enabling your networks ---------------------- > Travelping GmbH Phone: +49-391-81 90 99 0 > Roentgenstr. 13 Fax: +49-391-81 90 99 299 39108 > Magdeburg Email: info@REDACTED GERMANY > Web: http://www.travelping.com >> Company Registration: Amtsgericht Stendal Reg No.: HRB 10578> Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 > ---------------------------------------------------------------------> _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri May 4 19:18:20 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 4 May 2018 19:18:20 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Message-ID: I hope this hole fix please try it out: diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl index 0956d35..ed8663b 100644 --- a/lib/ssl/src/ssl_cipher.erl +++ b/lib/ssl/src/ssl_cipher.erl @@ -2837,11 +2837,13 @@ key_uses(OtpCert) -> Extensions = ssl_certificate:extensions_list(TBSExtensions), case ssl_certificate:select_extension(?'id-ce-keyUsage', Extensions) of undefined -> - undefined; + []; #'Extension'{extnValue = KeyUses} -> KeyUses end. +filter_keyuse_suites(_, [], CipherSuits, _) -> + CipherSuits; filter_keyuse_suites(Use, KeyUse, CipherSuits, Suites) -> case ssl_certificate:is_valid_key_usage(KeyUse, Use) of true -> Regards Ingela Erlang/OTP Team - Ericsson AB 2018-05-04 17:41 GMT+02:00 Ingela Andin : > Hi! > > Thank you for the example > > I did find one bug, the patch is here: > > diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl > index 0956d35..dd3dc54 100644 > --- a/lib/ssl/src/ssl_cipher.erl > +++ b/lib/ssl/src/ssl_cipher.erl > @@ -2837,7 +2837,7 @@ key_uses(OtpCert) -> > Extensions = ssl_certificate:extensions_list(TBSExtensions), > case ssl_certificate:select_extension(?'id-ce-keyUsage', Extensions) > of > undefined -> > - undefined; > + []; > #'Extension'{extnValue = KeyUses} -> > KeyUses > end. > > > my other sslv2 issue seems not to be related. (Probably a OpenSSL issue) > Your example does however not work perfect > so I am continuing to look into this! > > Regards Ingela Erlang/OTP team - Ericsson AB > > > > > > 2018-05-04 12:25 GMT+02:00 Roger Lipscombe : > >> On 4 May 2018 at 08:32, Ingela Andin wrote: >> > This error is consistent with one of the errors I am seeing in the >> nightly >> > builds when running OpenSSL with only default parameters so I suspect >> > something is off in combination >> > version negotiation and cipher suite selection checks. I am looking in >> to >> > it! >> >> I'm seeing the same, if it helps to reproduce. I generated my >> certificates with: >> >> #!/bin/sh >> >> # Create the CA key and certificate. >> openssl genrsa -out ca.key 2048 >> openssl req -new -x509 -nodes -key ca.key -days 3653 -out ca.pem -subj >> "/CN=Test CA" >> >> # Create the server key and CSR. >> openssl genrsa -out server.key 2048 >> openssl req -new -key server.key -out server.csr -subj "/CN=localhost" >> >> # Sign the CSR with the CA key. >> serial=$(date +"%s") >> openssl x509 -req -days 3563 -CA ca.pem -CAserial $serial >> -CAcreateserial -CAkey ca.key -in server.csr -out server.pem >> rm $serial >> >> I tested with: >> >> % Server >> {ok, _} = application:ensure_all_started(ssl). >> Port = 11002. >> LOpts = [{certfile, "server.pem"}, {keyfile, "server.key"}]. >> {ok, LSock} = ssl:listen(Port, LOpts). >> {ok, CSock} = ssl:transport_accept(LSock). >> ok = ssl:ssl_accept(CSock). >> >> % Client >> {ok, _} = application:ensure_all_started(ssl). >> Port = 11002. >> COpts = [{verify, verify_peer}, {cacertfile, "ca.pem"}]. >> {ok, Sock} = ssl:connect("localhost", Port, COpts). >> >> The server reports: >> >> =INFO REPORT==== 4-May-2018::11:22:20.971130 === >> TLS server: In state hello at tls_handshake.erl:130 generated SERVER >> ALERT: Fatal - Handshake Failure - malformed_handshake_data >> >> ** exception error: no match of right hand side value >> {error,{tls_alert,"handshake failure"}} >> >> The client reports: >> >> =INFO REPORT==== 4-May-2018::11:22:20.981524 === >> TLS client: In state hello received SERVER ALERT: Fatal - Handshake >> Failure >> >> ** exception error: no match of right hand side value >> {error,{tls_alert,"handshake failure"}} >> >> The same code works fine with 20.3.1 >> >> Thanks, >> Roger. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Fri May 4 19:21:05 2018 From: kostis@REDACTED (Kostis Sagonas) Date: Fri, 4 May 2018 19:21:05 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> Message-ID: <3e8af86e-5843-da23-8054-766697a1fa20@cs.ntua.gr> On 05/04/2018 04:58 PM, Andreas Schultz wrote: > Hi, > > The RC breaks rebar3 badly, not sure if it's rebar's fault or a problem > with the new logger... > > With an up to date rebar3 clone: > > ? ? rebar3$ ./bootstrap > ? (no logger present) unexpected logger message: {log,error,"Error in > process ~p with exit > value:~n~p~n",[<0.39.0>,{badarg,[{ets,lookup,[logger,proc_lib],[]},{logger_config,allow,3,[{file,"logger_config.erl"},{line,42}]},{proc_lib,crash_report,4,[{file,"proc_lib.erl"},{line,508}]},{proc_lib,exit_p,3,[{file,"proc_lib.erl"},{line,269}]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.37.0>,pid=><0.39.0>,time=>-576460751953591}} > ? (no logger present) unexpected logger message: {log,error,"Error in > process ~p with exit > value:~n~p~n",[<0.39.0>,{badarg,[{ets,lookup,[logger,proc_lib],[]},{logger_config,allow,3,[{file,"logger_config.erl"},{line,42}]},{proc_lib,crash_report,4,[{file,"proc_lib.erl"},{line,508}]},{proc_lib,exit_p,3,[{file,"proc_lib.erl"},{line,269}]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.37.0>,pid=><0.39.0>,time=>-576460751953591}} > ? ?... > > And so on... FWIW, I experienced this problem last Friday when I updated my copy of OTP's 'master' branch, and built starting from a 'make clean'. These errors were there with me all the time. Note that this has nothing to do with 'rebar*'. I was just trying to run the HiPE tests with ct_run. Then I spent about two hours investigating what the issue could be and did not really get anywhere. Then I tried a clean git checkout of OTP and that built a version that did not spit any such errors. Turns out that this issue can also be fixed by just zapping all generated files under 'bin' and then re-building everything: cd otp /bin/rm -r bin git checkout -- bin ./otp_build autoconf ; ./configure ... ; make // build again [I did not investigate further which exact file under bin was the culprit.] Perhaps the issue you are facing is similar and can be solved in a similar way. Kostis PS. A suggestion to the OTP folks: If there is no intention of fixing the 'make clean' target to work properly, why don't you just remove it so that there is less confusion? From michal@REDACTED Fri May 4 19:23:51 2018 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Fri, 4 May 2018 19:23:51 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <1525261079.4069.73.camel@ericsson.com> References: <1525261079.4069.73.camel@ericsson.com> Message-ID: <08a713f5-d1e0-4283-95ab-f59cff67185d@Spark> The release looks great. Thank you to the whole OTP team for the work you put into it. I wonder about the new file implementation that uses NIFs instead of ports, specifically the instrumentation around it. With ports, there was a way to list all ports and filter those that are files. This proved sometimes useful in debugging situations. Would it be possible to also get some kind of API to list all opened files, possibly with some additional information (like path)? I imagine a need for an API like that would be even more pressing once sockets are ported to use NIFs instead of ports as well. Micha?. On 2 May 2018, 13:38 +0200, Henrik Nord X , wrote: > OTP 21 Release Candidate 1 > > This is the first of two planned release candidates before the OTP 21 > release. The intention whit this release is that you as users try it > and give us feedback if something does not work as expected. Could be a > bug,an unexpected incompatibility, a significant change of > characteristics in a negative direction, etc. > > Erlang/OTP 21 is a new major release with new features, improvements as > well as incompatibilities. > > Potential Incompatibilities > > ?* All Corba applications are now moved from the OTP repository > ????* A new Corba repository will be created https://github.com/erlang > ?* New applications ftp and tftp, moved from inets > ?* ssl no longer supports 3_DES cipher suites or RSA-key exchange > cipher suites by default > ?* erlang:monitor on a primitive node (erl_interface, jinterface, etc) > will no longer fail with badarg exception. Instead a monitor will be > created, but it will only supervise the connection to the node. > > ?Highlights > > ?Erts: > > ????* Enhanced IO scalability > ????* Support for usage of distribution controller processes for > ??????alternative transports, routing etc > ????* compact instructions on 64bit systems for code below 4GB 20% less > ??????memory for loaded code > ????* Rewrite of the efile-driver with NIFs and "Dirty schedulers" > ??????resulting in faster file operations > ????* non-smp VM removed > ????* link and monitor optimized for scalability > ????* os:getenv/putenv now work on thread-safe emulation. No longer in > ??????sync with libc getenv(3). Manual synchronization will be needed. > > Compiler: > ????* Misc compiler optimizations including contributions from the > ??????Elixir team resulting in 10% improvements in benchmarks > ????* "Tuple calls" have been removed from the run-time system. > ????* Code such as f({ok, Val}) -> {ok, Val} is now automatically > ??????rewritten to f({ok, Val} = Tuple) -> Tuple. this reduces code > ??????size, execution time, and removed GC pressure. > ????* More information in stacktrace from a number of operators > ????* erlang:get_stacktrace/0 deprecated to be replaced with try ... > ??????catch C:R:Stacktrace -> ... > ????* Creation of small maps with literal keys optimized. > > Security: > ????* DTLS support in SSL > ????* Enhanced support for distribution over TLS > ????* "unsecure" ciphers removed from defaults in SSL and SSH. > ????* A new option value defined to facilitate implementing exec > ??????servers. Old option kept for compatibility, but now gives errors > ??????on stderror. > > Standard libraries: > ????* New API for logging, logger > ????* New uri_string module for parsing URIs according to "The > ??????standard" > ????* New function lists:search(list,fun/1) -> {ok, Value} | false > ????* Changed default behaviour of .erlang loading. escript, erlc, > ??????dialyzer and typer no longer load an .erlang at all. > > For more details see > http://erlang.org/download/otp_src_21.0-rc1.readme > > Pre built versions for Windows can be fetched here: > http://erlang.org/download/otp_win32_21.0-rc1.exe > http://erlang.org/download/otp_win64_21.0-rc1.exe > > Online documentation can be browsed here: > http://erlang.org/documentation/doc-10.0-rc1/doc/ > > The Erlang/OTP source can also be found at GitHub on the official > Erlang repository, > https://github.com/erlang/otp with tag OTP-21.0-rc1 > > > Thank you for all your contributions! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri May 4 23:37:31 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 4 May 2018 17:37:31 -0400 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <1525452943.3653629.1361100560.1798DB07@webmail.messagingengine.com> References: <1525261079.4069.73.camel@ericsson.com> <1525452943.3653629.1361100560.1798DB07@webmail.messagingengine.com> Message-ID: <20180504213730.GM1219@ferdmbp.local> On 05/04, Tristan Sloughter wrote: >We cut a new release today, 3.5.2, that builds with 21 fine. > Caveat: CT output is broken. You have to run `rebar3 ct --readable=false` The problem is that we still support R16 until 21 lands, and we can't support the logger API without support for maps since all of its messages are maps. I've made a branch of the app used to redirect CT Output at https://github.com/ferd/cth_readable/pull/18 From mark.geib.44@REDACTED Sun May 6 21:10:25 2018 From: mark.geib.44@REDACTED (Mark Geib) Date: Sun, 6 May 2018 12:10:25 -0700 Subject: [erlang-questions] Very long hang during OTP app startup Message-ID: Using Erlang R18 and rebar3. This is an application, or release, built with rebar3 and using generated management script to start and stop the application. I see very strange, and random behavior at startup in that when I start the application with "../ console" I will see erlang ?PROGRESS REPORT? messages as included applications start, like amps_client. It appears my OTP supervisor and children start normally. In fact one of the children opens a socket and handshakes with an external system. Once the handshake completes I expect to start receiving data from the external system, but nothing else happens until I see the following in the terminal where I started the application: =PROGRESS REPORT==== 4-May-2018 ::13:21:40 === supervisor: {local,kernel_safe_sup} started: [{pid,<0.563.0>}, {id,timer_server}, {mfargs,{timer,start_link,[]}}, {restart_type,permanent}, {shutdown,1000}, {child_type,worker}] At this point the app receives messages on the tcp socket and everything appears completely normal. However, sometimes it is hours before I see this last ?PROGRESS REPORT? and normal operation starts, other times maybe a minute or less. I have this feeling that a supervisor or child in another tree, not mine, is hung in an init() callback or something. I just am at a loss on how to debug this problem. I have never seen this type of behavior before. My release includes my application of course, but also lager, amqp_client and rabbit_common, and xmerl. Any suggestions would be greatly appreciated. Thanks, Mark Geib Principal Engineer Cheyenne Software Engineering mark.geib@REDACTED / 335-215 "A long habit of not thinking a thing wrong gives it a superficial appearance of being right.? -Thomas Paine -------------- next part -------------- An HTML attachment was scrubbed... URL: From torben.lehoff@REDACTED Sun May 6 22:46:41 2018 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Sun, 06 May 2018 20:46:41 +0000 Subject: [erlang-questions] Why doesn't gen_statem allow {next_event, ...} for state enter? In-Reply-To: <20180504133025.GA4698@erix.ericsson.se> References: <20180504133025.GA4698@erix.ericsson.se> Message-ID: Thx, time to upgrade! On Fri, 4 May 2018 at 15:31, Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > On Fri, May 04, 2018 at 12:35:25PM +0000, Torben Hoffmann wrote: > > Hi, > > > > I have just been bitten by this: > > state1(enter, ...) -> > > ... > > {next_state, state2, NewData, > > {next_event, internal, just_do_it}}; > > > > but that is not a legal action for an enter thingy. > > > > Why is that so? > > That bug was fixed in OTP-20.3.2. > > > > > > Cheers, > > Torben > > -- > > https://www.linkedin.com/in/torbenhoffmann/ > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- https://www.linkedin.com/in/torbenhoffmann/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Mon May 7 00:47:29 2018 From: g@REDACTED (Guilherme Andrade) Date: Sun, 6 May 2018 23:47:29 +0100 Subject: [erlang-questions] [ANN] aequitas: Fairness regulator and rate limiter Message-ID: Hello list, I'm pleased to announce the release of aequitas 1.0.0, a fairness regulator with optional rate limiting capabilities. It intends on allowing fair access to limited external resources, like databases and web services, amongst distinct actors. It does so by attempting to detect outliers in ordinary workload distributions. * Overview: https://github.com/g-andrade/aequitas/blob/master/README.md * Hex.pm package: https://hex.pm/packages/aequitas * Documentation: https://hexdocs.pm/aequitas/1.0.0/ * Source code: https://github.com/g-andrade/aequitas -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Mon May 7 08:34:30 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 7 May 2018 08:34:30 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <929A5E12-CDF9-442D-8EC3-5267CBCD2E91@gmx.de> Message-ID: Hi! 2018-05-04 14:40 GMT+02:00 Mahesh Paolini-Subramanya < mahesh@REDACTED>: > To echo Heinz's point, is there a *reason* for the deprecation? > > Enquiring minds want to know... > > > Th point is to phase out gen_fsm. The deprecated warning is important to get new users to choose gen_statem. Also if you have a living product you will probably be better of in the long run changing to gen_statem. This is not a complicated change and an exampel of how to do it is given in the gen_fsm manual page. Of course we give lots of time to adopt. We have no plan of removing the gen_fsm code swiftly, it is there in 20 and 21. I can see the deprecating process needs to enhanchend, and yes it sort of has more that one purpose which seems to conflict a bit. Regards Ingela Erlang/OTP team > > On Fri, May 4, 2018 at 8:29 AM, Oliver Korpilla > wrote: > >> Ah yes, dependencies! I see your problem now. >> >> We're using gen_statem for a test framework written in elixir for >> telecoms. The FSM has been the least of our problems. >> >> I'm always amazed at the range of Erlang versions still in use. I keep >> forgetting that. >> >> Oliver >> >> >> On May 4, 2018 8:25:35 AM EDT, "Heinz N. Gies" >> wrote: >>> >>> Hi Oliver, >>> I am not advocating to forbid migrating code to gen_statem when the >>> maintainer feels it is an improvement for their code. I?m advocating for >>> not forcing maintainers doing so for no reason. >>> >>> Karolis brought up another very good point I didn?t even think about. >>> Code written with gen_statem will not be able to run on older erlang >>> releases. While that?s probably not a issue on your own code where you >>> control the releases completely it?s a nightmare for open source and >>> libraries. >>> >>> >>> On 4. May 2018, at 14:12, Oliver Korpilla >>> wrote: >>> >>> Except for the init bug in 19.0 (which our infrastructure was fixed on >>> for a long time) we found migrating to gen_statem a painless experience >>> with decent improvements. But maybe we simply use a different or limited >>> feature set? >>> >>> Oliver >>> >>> On May 4, 2018 8:03:11 AM EDT, "Heinz N. Gies" >>> wrote: >>>> >>>> I have thought of that but I don?t think that?s a real option. >>>> Deprication warnings are important, this would turn all of them off which >>>> doesn?t help code quality. I want to know if something is deprecated and >>>> fix it in most cases. But as Karolis pointed out, it?s easy for a function >>>> or two but for a whole behaviour it?s nearly impossible. >>>> >>>> It basically creates two erlang worlds, pre-statem and post-statem. >>>> >>>> The stack trace is not great but it?s not as bad, it?s somewhat >>>> possible to just wrap the whole function calling it into a ifdef and call >>>> it a day. Not great it still is a decent amount of code duplication but >>>> it?s possible. https://github.com/Kyorai/clique/commit/b0e89de3cb >>>> b56396b59a1023903c7259ec2ef145#diff-7a19efbe2afedfb64aa7ec6690e9a13e would >>>> be an example. It?s more annoying then other changes but not nearly as bad >>>> as the gen_fsm >>>> >>>> On 4. May 2018, at 13:58, Roger Lipscombe >>>> wrote: >>>> >>>> On 4 May 2018 at 12:07, Karolis Petrauskas >>>> wrote: >>>> >>>>> I agree with Heinz. The deprecation of gen_fsm forced me to stuck on >>>>> OTP 19 and I dont see any reasonable path to upgrade. >>>>> >>>> >>>> We upgraded successfully to OTP 20 (and compiled against OTP 21) by >>>> simply adding the following to the top of our gen_fsm modules: >>>> >>>> -compile([nowarn_deprecated_function]). % gen_fsm is deprecated; use >>>> gen_statem >>>> >>>> It'll be a problem when gen_fsm is *removed*, but until then, we'll >>>> stick with this. >>>> >>>> >>>> >>> -- >>> Sent from my Android device with K-9 Mail. Please excuse my brevity. >>> >>> >>> >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > > *Mahesh Paolini-Subramanya > That > tall bald Indian guy..* > *Twitter | Blog > | G+ > | LinkedIn > * > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Mon May 7 08:54:50 2018 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Mon, 07 May 2018 06:54:50 +0000 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: What about documentation from the OTP team that outlines a deprecation policy? It may be a good opportunity to also outline other compatibility guarantees, such as the compiled bytecode guarantees and the node compatibility guarantees, if such are not yet documented. We did define such document not long ago for Elixir. In terms of deprecations, we have two terms: "soft-deprecation" and "hard-deprecation". The soft-deprecation is added as soon as a new implementation exists. At this point we don't emit any warnings, but we do update the docs to say the feature will warn in the future and start to point folks towards better ways. A hard-deprecation is when we finally start emitting warnings. A hard-deprecation can only be added after an alternative exists for at least 2 Elixir releases. This is very important because it means that, once a feature is hard deprecated, you know you can use the proposed alternative and that alternative is supported at least two versions back. This is only necessary if a feature is being replaced. In case a feature is being removed, such as non-smp VM versions, you may skip directly to the "hard-deprecation" and emit warnings straight-away since there won?t be an alternative in the future anyway. I am not proposing for Erlang/OTP team to follow those rules and conventions but writing *some* rules as minimum guarantees may improve communication and help the community plan in terms of warnings, deprecations and removals accordingly. I would just avoid emitting deprecation warnings in the same version that an alternative is introduced, because it means library developers need to either only support the latest version or introduce conditionals (either at compile-time or runtime) to support multiple versions. *Jos? Valimwww.plataformatec.com.br Founder and Director of R&D* -- *Jos? Valimwww.plataformatec.com.br Founder and Director of R&D* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Mon May 7 08:55:05 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 7 May 2018 08:55:05 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Message-ID: Hi again! I think I need to take back that last change, it is a bit more complicated. I will work on a real fix during the week. Regards Ingela Erlang/OTP Team 2018-05-04 19:18 GMT+02:00 Ingela Andin : > I hope this hole fix please try it out: > > diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl > index 0956d35..ed8663b 100644 > --- a/lib/ssl/src/ssl_cipher.erl > +++ b/lib/ssl/src/ssl_cipher.erl > @@ -2837,11 +2837,13 @@ key_uses(OtpCert) -> > Extensions = ssl_certificate:extensions_list(TBSExtensions), > case ssl_certificate:select_extension(?'id-ce-keyUsage', Extensions) > of > undefined -> > - undefined; > + []; > #'Extension'{extnValue = KeyUses} -> > KeyUses > end. > > +filter_keyuse_suites(_, [], CipherSuits, _) -> > + CipherSuits; > filter_keyuse_suites(Use, KeyUse, CipherSuits, Suites) -> > case ssl_certificate:is_valid_key_usage(KeyUse, Use) of > true -> > > > Regards Ingela Erlang/OTP Team - Ericsson AB > > > > 2018-05-04 17:41 GMT+02:00 Ingela Andin : > >> Hi! >> >> Thank you for the example >> >> I did find one bug, the patch is here: >> >> diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl >> index 0956d35..dd3dc54 100644 >> --- a/lib/ssl/src/ssl_cipher.erl >> +++ b/lib/ssl/src/ssl_cipher.erl >> @@ -2837,7 +2837,7 @@ key_uses(OtpCert) -> >> Extensions = ssl_certificate:extensions_list(TBSExtensions), >> case ssl_certificate:select_extension(?'id-ce-keyUsage', >> Extensions) of >> undefined -> >> - undefined; >> + []; >> #'Extension'{extnValue = KeyUses} -> >> KeyUses >> end. >> >> >> my other sslv2 issue seems not to be related. (Probably a OpenSSL issue) >> Your example does however not work perfect >> so I am continuing to look into this! >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> >> >> >> 2018-05-04 12:25 GMT+02:00 Roger Lipscombe : >> >>> On 4 May 2018 at 08:32, Ingela Andin wrote: >>> > This error is consistent with one of the errors I am seeing in the >>> nightly >>> > builds when running OpenSSL with only default parameters so I suspect >>> > something is off in combination >>> > version negotiation and cipher suite selection checks. I am looking in >>> to >>> > it! >>> >>> I'm seeing the same, if it helps to reproduce. I generated my >>> certificates with: >>> >>> #!/bin/sh >>> >>> # Create the CA key and certificate. >>> openssl genrsa -out ca.key 2048 >>> openssl req -new -x509 -nodes -key ca.key -days 3653 -out ca.pem -subj >>> "/CN=Test CA" >>> >>> # Create the server key and CSR. >>> openssl genrsa -out server.key 2048 >>> openssl req -new -key server.key -out server.csr -subj "/CN=localhost" >>> >>> # Sign the CSR with the CA key. >>> serial=$(date +"%s") >>> openssl x509 -req -days 3563 -CA ca.pem -CAserial $serial >>> -CAcreateserial -CAkey ca.key -in server.csr -out server.pem >>> rm $serial >>> >>> I tested with: >>> >>> % Server >>> {ok, _} = application:ensure_all_started(ssl). >>> Port = 11002. >>> LOpts = [{certfile, "server.pem"}, {keyfile, "server.key"}]. >>> {ok, LSock} = ssl:listen(Port, LOpts). >>> {ok, CSock} = ssl:transport_accept(LSock). >>> ok = ssl:ssl_accept(CSock). >>> >>> % Client >>> {ok, _} = application:ensure_all_started(ssl). >>> Port = 11002. >>> COpts = [{verify, verify_peer}, {cacertfile, "ca.pem"}]. >>> {ok, Sock} = ssl:connect("localhost", Port, COpts). >>> >>> The server reports: >>> >>> =INFO REPORT==== 4-May-2018::11:22:20.971130 === >>> TLS server: In state hello at tls_handshake.erl:130 generated SERVER >>> ALERT: Fatal - Handshake Failure - malformed_handshake_data >>> >>> ** exception error: no match of right hand side value >>> {error,{tls_alert,"handshake failure"}} >>> >>> The client reports: >>> >>> =INFO REPORT==== 4-May-2018::11:22:20.981524 === >>> TLS client: In state hello received SERVER ALERT: Fatal - Handshake >>> Failure >>> >>> ** exception error: no match of right hand side value >>> {error,{tls_alert,"handshake failure"}} >>> >>> The same code works fine with 20.3.1 >>> >>> Thanks, >>> Roger. >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Mon May 7 09:35:23 2018 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 7 May 2018 00:35:23 -0700 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: <952C4508-9679-41C5-89A2-967279EA5FFB@licenser.net> <929A5E12-CDF9-442D-8EC3-5267CBCD2E91@gmx.de> Message-ID: <788016d7-aca7-8208-50c7-8f8f99379fca@gmail.com> On 05/06/2018 11:34 PM, Ingela Andin wrote: > Hi! > > 2018-05-04 14:40 GMT+02:00 Mahesh Paolini-Subramanya >: > > To echo Heinz's point, is there a *reason* for the deprecation? > > Enquiring minds want to know... > > > Th point is to phase out gen_fsm. The deprecated warning is important to get new users to choose gen_statem. Also if you have a living product you will probably be better of in the long run changing to gen_statem. This > is not a complicated change and an exampel of how to do it is given in the gen_fsm manual page. Of course we give lots of time to adopt. We have no plan of removing the gen_fsm code swiftly, it is there in 20 and 21. > I can see the deprecating process needs to enhanchend, and yes it sort of has more that one purpose which seems to conflict a bit. One thing that might help is if we have the ability to deprecate types which I entered an entry for at https://bugs.erlang.org/browse/ERL-335 .? For example,? when all the time units switched from a format like milli_seconds to millisecond in Erlang/OTP 20.0 it would have been easier to have a deprecated type that can be used for an easier to understand warning.? That may seem like a small issue that can be avoided by switching to other functions or modules, but often changes don't need to be large for small changes that can be limited to type changes. Best Regards, Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon May 7 09:59:54 2018 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 7 May 2018 09:59:54 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: <08a713f5-d1e0-4283-95ab-f59cff67185d@Spark> References: <1525261079.4069.73.camel@ericsson.com> <08a713f5-d1e0-4283-95ab-f59cff67185d@Spark> Message-ID: Hello, On Fri, May 4, 2018 at 7:23 PM, Micha? Muska?a wrote: > The release looks great. Thank you to the whole OTP team for the work you > put into it. > Thanks! > > I wonder about the new file implementation that uses NIFs instead of > ports, specifically the instrumentation around it. With ports, there was a > way to list all ports and filter those that are files. This proved > sometimes useful in debugging situations. Would it be possible to also get > some kind of API to list all opened files, possibly with some additional > information (like path)? > I imagine a need for an API like that would be even more pressing once > sockets are ported to use NIFs instead of ports as well. > Yes, I also believe that something like this is going to be needed. While it is possible on most OSs to look in the /proc/ fs to see which files are opened, it is definitely useful to be able to track back to the process controlling it. Even more so for sockets. Personally I would prefer not to have to build something new within erts to handle this, but instead use what we have. Maybe ets tables that keeps track of files, connections, etc. Each sockets will most likely have to be associated with a process, so for them it could be possible to just store something in the process dictionary and let the debug process get the pdict of all processes. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Mon May 7 10:54:06 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 7 May 2018 10:54:06 +0200 Subject: [erlang-questions] gen_server and exit(Pid, shutdown) In-Reply-To: References: Message-ID: <20180507085406.GA83740@erix.ericsson.se> On Fri, May 04, 2018 at 07:23:23PM +0300, Alexander Petrovsky wrote: > Hello! > > After a long timer working with Erlang and using gen_server abstraction, > I've discovered for myself that when I send exit(Pid, shutdown) for some > gen_server (or when application stopping), according to the documentation, > terminate callback will be called only when trap_exit flag is true. In my > mind it's absolutely not intuitively, the documentation very clear > for trap_exit and it's says: In http://erlang.org/doc/man/gen_server.html, Description, 5:th paragraph: Notice that a gen_server process does not trap exit signals automatically, this must be explicitly initiated in the callback module. In http://erlang.org/doc/man/gen_server.html#Module:terminate-2, 4:th paragraph, first point in bullet list: * The gen_server process has been set to trap exit signals. This is a process property, it applies to all processes. In http://erlang.org/doc/reference_manual/processes.html#id88677, section 12.7 Error Handling, Receiving Exit Signals: The default behaviour when a process receives an exit signal with an exit reason other than normal, is to terminate and in turn emit exit signals with the same exit reason to its linked processes. An exit signal with reason normal is ignored. A process can be set to trap exit signals by calling: process_flag(trap_exit, true) When a process is trapping exits, it does not terminate when an exit signal is received. Instead, the signal is transformed into a message {'EXIT',FromPid,Reason}, which is put into the mailbox of the process, just like a regular message. An exception to the above is if the exit reason is kill, that is if exit(Pid,kill) has been called. This unconditionally terminates the process, regardless of if it is trapping exit signals. > > When a process is trapping exits, it does not terminate when an exit signal > > is received. Instead, the signal is transformed into a message > > {'EXIT',FromPid,Reason}, which is put into the mailbox of the process, > > just like a regular message. > > > So, I suppose to see the message {'EXIT', FromPid, shutdown} in > handle_info when > I call exit(GenServerPid, shutdown), and I always suppose, that terminate > callback will be called (of course I remeber about reason kill) I do not see a question here, is there one? In http://erlang.org/doc/man/gen_server.html#Module:terminate-2 Even if the gen_server process is not part of a supervision tree, this function is called if it receives an 'EXIT' message from its parent. Reason is the same as in the 'EXIT' message. So, the gen_server will interpret an {'EXIT',Parent,Reason} as an order to terminate. Note, that it has to be from the Parent, that is; the process that started the gen_server. Any other exit message is handled in handle_info/2 just like any other process message. And if the process does not trap exits it is like any other Erlang process killed by exit(Pid, Reason). > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 931 9877991 -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From mfrench@REDACTED Mon May 7 10:13:49 2018 From: mfrench@REDACTED (Mike French) Date: Mon, 7 May 2018 08:13:49 +0000 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: <9c2d5028636249ff807308621f935ab5@TCC-EXCH-01.tcclimited1.com> So you have - ?soft-deprecation? with doc notes, but no compiler warnings - ?hard-deprecation? with compiler warnings, but no feature removal I suggest the third stage would be: - ?end-of-life? with compiler warnings, and there is a known future release when the original feature will vanish Mike From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Jos? Valim Sent: Monday, May 07, 2018 9:55 AM To: Lloyd R. Prentice Cc: erlang-questions@REDACTED Questions Subject: Re: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine What about documentation from the OTP team that outlines a deprecation policy? It may be a good opportunity to also outline other compatibility guarantees, such as the compiled bytecode guarantees and the node compatibility guarantees, if such are not yet documented. We did define such document not long ago for Elixir. In terms of deprecations, we have two terms: "soft-deprecation" and "hard-deprecation". The soft-deprecation is added as soon as a new implementation exists. At this point we don't emit any warnings, but we do update the docs to say the feature will warn in the future and start to point folks towards better ways. A hard-deprecation is when we finally start emitting warnings. A hard-deprecation can only be added after an alternative exists for at least 2 Elixir releases. This is very important because it means that, once a feature is hard deprecated, you know you can use the proposed alternative and that alternative is supported at least two versions back. This is only necessary if a feature is being replaced. In case a feature is being removed, such as non-smp VM versions, you may skip directly to the "hard-deprecation" and emit warnings straight-away since there won?t be an alternative in the future anyway. I am not proposing for Erlang/OTP team to follow those rules and conventions but writing *some* rules as minimum guarantees may improve communication and help the community plan in terms of warnings, deprecations and removals accordingly. I would just avoid emitting deprecation warnings in the same version that an alternative is introduced, because it means library developers need to either only support the latest version or introduce conditionals (either at compile-time or runtime) to support multiple versions. Jos? Valim www.plataformatec.com.br Founder and Director of R&D -- Jos? Valim www.plataformatec.com.br Founder and Director of R&D -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon May 7 11:21:43 2018 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 7 May 2018 11:21:43 +0200 Subject: [erlang-questions] How can we increase multiblock carrier utilization for binary_alloc? In-Reply-To: References: Message-ID: On Thu, May 3, 2018 at 12:18 PM, Gerhard Lazu wrote: > Hi Lukas, > > Why do you not use erlang:memory() as the base for whether you can accept >> more messages? Having a low memory utilisation is not bad in itself, unless >> of course some other program on the same machine needs the memory. >> > > We used to use erlang:memory(), but we've learned that it doesn't work > well in practice [1]. Linux OOM will take action based on RSS, not Erlang > allocated memory. > Yes, looking at erlang:memory() could make you end up in those scenarios. However it should be possible to look at recon_alloc:memory(unused), to get a ballpark figure about how much memory that the Erlang VM has reserved that it is not using at the moment. > > >> Looking at the used memory under load and after, at peak the allocated >> memory is 1510 MB and then after it is 577 MB. So about 2/3rds of the >> allocated memory was returned to the OS. While this is not perfect, it is >> not terrible either. Reducing it further may not be easy. >> > > Your observation is true and accurate. It's also true and accurate that > out of 577MB allocated, 300MB is used & 277MB is unused, meaning that > almost half of the allocated memory is not used. > > I understand that it may not be easy to reduce the unused memory, but all > I'm thinking is that whilst this unused memory might seem small in this > particular scenario, what happens when the Erlang VM has 60GB allocated? > One thing that you could try is to see if malloc does a better job than erts does. In general the erts allocators are better at scalability, while malloc is better at performance. I don't know which is better at dealing with fragmentation, use "+MBe false" to disable it for binary alloc. The really bad part about this is that you loose all statistics. So if you do get into other issues it will be much harder to figure out what is going on. You could also try to play with the sbct to see if you get better allocations using a lower value. This will cause more allocations to be places into sbcs which could be good for fragmentation. You seem to have an average block size of about 2 kb, so I would try setting the sbct to about double that to start with and see if you notice any difference. It's hard to know what be a good value with the OTP-20 instrumentation, so you will have to try and see if you get any difference. > Would it help if we can show the impact of this behaviour on hosts with > larger memory usage? > No I don't think so. The behavior should be similar just with scaled up values. I suppose that it would be a good idea to verify that if you can. > > >> Can you reproduce the behaviour? Would it be possible to get a >> recon_alloc snaphot during and after load? >> > > I'm sharing recon_alloc snapshots during & after load for the following: > > 1. erts_alloc defaults (lmbcs 5120) [2] > 2. -MBlmbcs 512 [3] > > I've also captured a during & after load screenshots of the 2 > configurations running side-by-side (left is erts_alloc defaults (lmbcs > 5120), right is -MBlmbcs 512) [4]. > > While our initial configuration used a few more flags, -MHas ageffcbf > -MBas ageffcbf -MHlmbcs 512 -MBlmbcs 512, I've kept things as simple as > possible on this run, and only used -MBlmbcs 512 > > Have you seen https://github.com/erlang/otp/pull/1790 that was just >> merged to master with the accompanying blog post: >> http://blog.erlang.org/Memory-instrumentation-in-OTP-21/? >> > > I haven't, thank you for sharing. We are waiting on Elixir #6611 before we > can test against OTP 21.0-rc1 [5]. > > >> I assume that you have tried ageffcaobf? >> > > Yes, we have tried all allocation strategies. ageffcbf resulted in > "spikier" CPU and dirty mem writeback, but also sharper drops in dirty mem > writeback. Under load, ageffcbf had 1% lower RSS usage, and 2.5% lower > unused memory than ageffcaobf. After load however, ageffcbf had 5% lower > RSS usage & 4% lower unused memory. In conclusion, ageffcbf proved the > best out of all allocation strategies. > > Here is a side-by-side comparison of -MBas ageffcaobf -MBlmbcs 512 (left) > vs -MBas ageffcbf -MBlmbcs 512 (right) [6], and the relevant recon_alloc > snapshots [7]. > While aobf should be a little bit better, I suspect that since we introduced the carrier pool and thus use the the carrier strategies the difference is within the error margins, especially when you run a system with lots of small carriers. On a related point, this discussion has caused me to start looking at using madvise/VirtualAlloc to let the OS know that pages within carriers are not used by erts any more. I'm not sure how that will interact with RSS as from I I've been able to figure out the pages remain associated with the program until the OS needs them so satisfy some memory request. Any such feature won't be in OTP-21, but may be added later. > Thank you Lukas for helping out with this, Gerhard. > > [1] https://github.com/rabbitmq/rabbitmq-server/issues/1223 & > https://github.com/rabbitmq/rabbitmq-server/pull/1259#issu > ecomment-308428057 - the entire PR context is valuable and relevant > [2] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory > -allocators/during-MBlmbcs_5120.recon_alloc.snapshot & > https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory- > allocators/after-MBlmbcs_5120.recon_alloc.snapshot > [3] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory > -allocators/during-MBlmbcs_512.recon_alloc.snapshot & > https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory- > allocators/after-MBlmbcs_512.recon_alloc.snapshot > [4] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory > -allocators/during-MBlmbcs_5120-vs-MBlmbcs_512.png & > https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory- > allocators/after-MBlmbcs_5120-vs-MBlmbcs_512.png > [5] *https://github.com/elixir-lang/elixir/issues/6611#issuecomment-386208496 > * > [6] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory > -allocators/during-MBas_ageffcaobf-MBlmbcs_512.recon_alloc.snapshot & > https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory- > allocators/after-MBas_ageffcaobf-MBlmbcs_512.recon_alloc.snapshot + > https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory- > allocators/during-MBas_ageffcbf-MBlmbcs-512.recon_alloc.snapshot & > https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory- > allocators/after-MBas_ageffcbf-MBlmbcs-512.recon_alloc.snapshot > [7] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory > -allocators/during-MBas_ageffcaobf-MBlmbcs_512-vs-MBas_ > ageffcbf-MBlmbcs_512.png & https://s3-eu-west-1.amazona > ws.com/rabbitmq-share/memory-allocators/after-MBas_ageffcao > bf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png + > https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory- > allocators/during-cpu-MBas_ageffcaobf-MBlmbcs_512-vs- > MBas_ageffcbf-MBlmbcs_512.png & https://s3-eu-west-1.amazonaws > .com/rabbitmq-share/memory-allocators/during-memory-MBas_ > ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth@REDACTED Mon May 7 12:06:37 2018 From: kenneth@REDACTED (Kenneth Lundin) Date: Mon, 7 May 2018 12:06:37 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine Message-ID: On Mon, May 7, 2018 at 8:54 AM, Jos? Valim wrote: > > What about documentation from the OTP team that outlines a deprecation > policy? It may be a good opportunity to also outline other compatibility > guarantees, such as the compiled bytecode guarantees and the node > compatibility guarantees, if such are not yet documented. > I think it is a good idea to describe the OTP policy regarding deprecation, backward compatibility, supported releases etc. in one place. Will try to have it available before the OTP 21 release. The Elixir policy with hard and soft deperecation is also interesting as we need to make it clearer what a depercation means. But in short we use the deprecation mechanism to flag that a function och application is no longer recommended for use in new code since there exist another solution that we recomment to be used instead. We don't remove deprecated functions or applications just like that and some functions have to stay for a very long time (or forever) since there is so much usage of them. gen_fsm is an old module which is used a lot and it is not likely to dissapear in a near future. I realize that the deprecated attribute inside the module like this -deprecated({start, 3, next_major_release}). is somewhat misleading and we will try to improve this. /Kenneth, Erlang/OTP > > We did define such document not long ago for Elixir. In terms of > deprecations, we have two terms: "soft-deprecation" and "hard-deprecation". > The soft-deprecation is added as soon as a new implementation exists. At > this point we don't emit any warnings, but we do update the docs to say the > feature will warn in the future and start to point folks towards better > ways. > > A hard-deprecation is when we finally start emitting warnings. A > hard-deprecation can only be added after an alternative exists for at least > 2 Elixir releases. This is very important because it means that, once a > feature is hard deprecated, you know you can use the proposed alternative > and that alternative is supported at least two versions back. > > This is only necessary if a feature is being replaced. In case a feature > is being removed, such as non-smp VM versions, you may skip directly to the > "hard-deprecation" and emit warnings straight-away since there won?t be an > alternative in the future anyway. > > I am not proposing for Erlang/OTP team to follow those rules and > conventions but writing *some* rules as minimum guarantees may improve > communication and help the community plan in terms of warnings, > deprecations and removals accordingly. I would just avoid emitting > deprecation warnings in the same version that an alternative is introduced, > because it means library developers need to either only support the latest > version or introduce conditionals (either at compile-time or runtime) to > support multiple versions. > > *Jos? Valimwww.plataformatec.com.br > Founder and Director of R&D* > > -- > > > *Jos? Valimwww.plataformatec.com.br > Founder and Director of R&D* > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Mon May 7 15:45:22 2018 From: codewiget95@REDACTED (asdf asdf) Date: Mon, 7 May 2018 09:45:22 -0400 Subject: [erlang-questions] Deployment rebar3 Message-ID: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> Hey everyone, I?ve recently built a few applications that handle high volume message passing and all communicate with each other using Erlang?s ssl distribution. I?m working on deploying these to multiple servers - right now I am just using rebar3 release and creating a zipped tar of the release directory, moving it to a new box, and changing the some vm.args and sys.config options on the new machines. This doesn?t seem totally intuitive though? it take far longer than I feel like it should. I?ve read about Docker and Kubernetes, and those seem promising, but our IT guys aren?t ready to move to containers and are using static nodes right now. So, what is the best way to quickly distribute your erlang binaries and configure the enviornment? Have a quick script to do it automatically? Is the answer just containerize with Docker and wait a few months for the IT team to catch up with it? Or is there some other option out there? Thanks for your help! -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Mon May 7 16:03:45 2018 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Mon, 7 May 2018 16:03:45 +0200 Subject: [erlang-questions] Deployment rebar3 In-Reply-To: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> References: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> Message-ID: What's "best" is very much depending on who you ask, but, for starters you may consider wrapping your configuration with something that allows you to override config variables with environment variables (see example below). This would make it easier to run things without changing files and it would also prepare you for whenever you enter docker-/kube-land later also...should you decide to take that step. /Stefan ...an example -module(config). -export([get/1]). %% reads a configuration value Key. Values are placed in sys.config as atoms %% but can be overridden by setting environment variables in the os. The environment %% variable name is given by the atom name converted to a string/list and then uppercased %% example: my_atom_key would translate to the environment variable MY_ATOM_KEY get(Key) when is_atom(Key) -> {ok, Default} = application:get_env(axon, Key), EnvName = string:to_upper(atom_to_list(Key)), case os:getenv(EnvName) of false -> Default; %% assume it has the right type in config OSVal -> convert(OSVal, Default) end. convert(Val, Default) when is_list(Val), is_integer(Default) -> list_to_integer(Val); convert(Val, Default) when is_list(Val), is_binary(Default) -> list_to_binary(Val); convert(Val, Default) when is_list(Val), is_boolean(Default) -> list_to_atom(Val); convert(Val, Default) when is_list(Val), is_list(Default) -> Val. On Mon, May 7, 2018 at 3:45 PM, asdf asdf wrote: > Hey everyone, > > I?ve recently built a few applications that handle high volume message > passing and all communicate with each other using Erlang?s ssl > distribution. I?m working on deploying these to multiple servers - right > now I am just using rebar3 release and creating a zipped tar of the release > directory, moving it to a new box, and changing the some vm.args and > sys.config options on the new machines. This doesn?t seem totally intuitive > though? it take far longer than I feel like it should. > > I?ve read about Docker and Kubernetes, and those seem promising, but our > IT guys aren?t ready to move to containers and are using static nodes right > now. > > So, what is the best way to quickly distribute your erlang binaries and > configure the enviornment? Have a quick script to do it automatically? Is > the answer just containerize with Docker and wait a few months for the IT > team to catch up with it? Or is there some other option out there? > > Thanks for your help! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Mon May 7 16:44:26 2018 From: g@REDACTED (Guilherme Andrade) Date: Mon, 7 May 2018 15:44:26 +0100 Subject: [erlang-questions] Deployment rebar3 In-Reply-To: References: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> Message-ID: On 7 May 2018 at 15:03, Stefan Hellkvist wrote: > What's "best" is very much depending on who you ask, but, for starters you > may consider wrapping your configuration with something that allows you to > override config variables with environment variables (see example below). > rebar3 releases provide this out of the box: https://www.rebar3.org/docs/releases#dynamic-configuration -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Mon May 7 16:51:50 2018 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Mon, 7 May 2018 16:51:50 +0200 Subject: [erlang-questions] Deployment rebar3 In-Reply-To: References: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> Message-ID: <212BA38A-3758-4E9C-9E4D-CFADFC26C22F@gmail.com> > 7 maj 2018 kl. 16:44 skrev Guilherme Andrade : > > > >> On 7 May 2018 at 15:03, Stefan Hellkvist wrote: >> What's "best" is very much depending on who you ask, but, for starters you may consider wrapping your configuration with something that allows you to override config variables with environment variables (see example below). > > rebar3 releases provide this out of the box: > > https://www.rebar3.org/docs/releases#dynamic-configuration It is a relx feature, yes, which works fine also in erlang.mk, but I find it sometimes inconvenient that you then HAVE to set the environment variable every time. But, agreed, it is also a good option if you start your release with a script. Stefan -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Mon May 7 17:16:43 2018 From: codewiget95@REDACTED (asdf asdf) Date: Mon, 7 May 2018 11:16:43 -0400 Subject: [erlang-questions] Deployment rebar3 In-Reply-To: <212BA38A-3758-4E9C-9E4D-CFADFC26C22F@gmail.com> References: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> <212BA38A-3758-4E9C-9E4D-CFADFC26C22F@gmail.com> Message-ID: <6d78130b-3c86-4e05-8d4f-c943a48aebe5@Spark> Using OS env variables would definitely speed up the process. I could have a ?master? config file that is exported to the ENV and then each node would get all of the important info. Really, the changing information that just got annoying was entering the node name and IP address for the local network and the public IP addresses to listen on for certain information. This could certainly hold us over until we move into containerized systems. Thanks! On May 7, 2018, 10:51 AM -0400, Stefan Hellkvist , wrote: > > > 7 maj 2018 kl. 16:44 skrev Guilherme Andrade : > > > > > > > > On 7 May 2018 at 15:03, Stefan Hellkvist wrote: > > > > What's "best" is very much depending on who you ask, but, for starters you may consider wrapping your configuration with something that allows you to override config variables with environment variables (see example below). > > > > > > rebar3 releases provide this out of the box: > > > > > > https://www.rebar3.org/docs/releases#dynamic-configuration > > It is a relx feature, yes, which works fine also in erlang.mk, but I find it sometimes inconvenient that you then HAVE to set the environment variable every time. But, agreed, it is also a good option if you start your release with a script. > > Stefan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Mon May 7 17:57:42 2018 From: t@REDACTED (Tristan Sloughter) Date: Mon, 07 May 2018 08:57:42 -0700 Subject: [erlang-questions] Deployment rebar3 In-Reply-To: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> References: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> Message-ID: <1525708662.1367804.1363617360.4BFE90BA@webmail.messagingengine.com> Note that you, assuming you aren't already, should use `rebar3 tar` and not tar/zip the release directory manually. How to deploy a release is more of a general problem. You should be able to use whatever your IT guys are already using for copying a release to a node and setting up configuration -- ansible, chef, etc? You can use either OS vars as someone pointed out or simply have one of those configuration tools create a whole new sys.config/vm.args and overwrite the existing files after unpacking the release. Docker doesn't really bring you much if not also using something like kubernetes. You still have to get the image to a node and setup per-node configuration, not much different from doing it with the tarred release. But kubernetes is really great for this :). I use kubernetes deployments and stateful sets to deploy Erlang releases, with env vars for configuring sys.config/vm.args per pod and kubernetes dns for node discovery/clustering. Tristan On Mon, May 7, 2018, at 6:45 AM, asdf asdf wrote: > Hey everyone, > > I?ve recently built a few applications that handle high volume > message passing and all communicate with each other using Erlang?s > ssl distribution. I?m working on deploying these to multiple servers > - right now I am just using rebar3 release and creating a zipped tar > of the release directory, moving it to a new box, and changing the > some vm.args and sys.config options on the new machines. This > doesn?t seem totally intuitive though? it take far longer than I > feel like it should.> > I?ve read about Docker and Kubernetes, and those seem promising, but > our IT guys aren?t ready to move to containers and are using static > nodes right now.> > So, what is the best way to quickly distribute your erlang binaries > and configure the enviornment? Have a quick script to do it > automatically? Is the answer just containerize with Docker and wait a > few months for the IT team to catch up with it? Or is there some other > option out there?> > Thanks for your help! > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Tue May 8 11:55:52 2018 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Tue, 8 May 2018 11:55:52 +0200 Subject: [erlang-questions] Patch package OTP 20.3.6 released Message-ID: <90412db7-1f5f-41a9-b98b-a1b59bf1a2c6@ericsson.com> Patch Package: OTP 20.3.6 Git Tag: OTP-20.3.6 Date: 2018-05-08 Trouble Report Id: OTP-15064, OTP-15066, OTP-15073, OTP-15074 Seq num: ERL-618 System: OTP Release: 20 Application: crypto-4.2.2, ssh-4.6.9 Predecessor: OTP 20.3.5 Check out the git tag OTP-20.3.6, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- crypto-4.2.2 ---------------------------------------------------- --------------------------------------------------------------------- The crypto-4.2.2 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-15073 Application(s): crypto If OPENSSL_NO_EC was set, the compilation of the crypto nifs failed. OTP-15074 Application(s): crypto Related Id(s): ERL-618 C-compile errors for LibreSSL 2.7.0 - 2.7.2 fixed Full runtime dependencies of crypto-4.2.2: erts-9.0, kernel-5.3, stdlib-3.4 --------------------------------------------------------------------- --- ssh-4.6.9 ------------------------------------------------------- --------------------------------------------------------------------- Note! The ssh-4.6.9 application can *not* be applied independently of other applications on an arbitrary OTP 20 installation. On a full OTP 20 installation, also the following runtime dependencies have to be satisfied: -- crypto-4.2 (first satisfied in OTP 20.2) -- public_key-1.5.2 (first satisfied in OTP 20.2) --- Fixed Bugs and Malfunctions --- OTP-15064 Application(s): ssh Host key hash erroneously calculated for clients following draft-00 of RFC 4419, for example PuTTY OTP-15066 Application(s): ssh Renegotiation could fail in some states Full runtime dependencies of ssh-4.6.9: crypto-4.2, erts-6.0, kernel-3.0, public_key-1.5.2, stdlib-3.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From valerii.tikhonov@REDACTED Tue May 8 13:54:19 2018 From: valerii.tikhonov@REDACTED (Valery Tikhonov) Date: Tue, 8 May 2018 13:54:19 +0200 Subject: [erlang-questions] (no subject) Message-ID: > Hey everyone, > > > > > I?ve recently built a few applications that handle high volume message passing and all communicate with each other using Erlang?s ssl distribution. I?m working on deploying these to multiple servers - right now I am just using rebar3 release and creating a zipped tar of the release directory, moving it to a new box, and changing the some vm.args and sys.config options on the new machines. This doesn?t seem totally intuitive though? it take far longer than I feel like it should. > > > > > I?ve read about Docker and Kubernetes, and those seem promising, but our IT guys aren?t ready to move to containers and are using static nodes right now. > > > > > So, what is the best way to quickly distribute your erlang binaries and configure the enviornment? Have a quick script to do it automatically? Is the answer just containerize with Docker and wait a few months for the IT team to catch up with it? Or is there some other option out there? > > > > > Thanks for your help! Please take a look at Enot (https://github.com/comtihon/enot), this is actually I've created it for. It can help you with both deployment and environment variables handling. For deployment - just add install steps you perform to enot's config (here is an example https://github.com/comtihon/example_service) and you will be able to install and run it on another linux machine with `enot install you/yourservice`. If your service is private (not open sourced) - you should run your enot auto builder first, to store enot packages there ( https://github.com/comtihon/enot_auto_builder). For templating - you can use environment variables in app.src, relx.conf or vm.args: {env, [ {% if MAGIC_GREETING is defined %} {greeting, "{{ MAGIC_GREETING }}"} {% else %} {greeting, "hello world!" } {% endif %} ]} or: {% if app.git_branch == 'master' %} {log, info} {% else %} {log, debug} {% endif %} or: {% if hostname in prod_hosts %} {database, [ {host, "prod_db"}, {password, "super_secure_password"}, {pool_size, 400} ]} {% elif hostname in stage_hosts %} {database, [ {host, "staging_db"}, {password, "other_secure_password"}, {pool_size, 200} ]} {% else %} {database, [ {host, "dev_host"}, {password, "dev_password"}, {pool_size, 10} ]} {% endif %} Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue May 8 14:07:22 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 08 May 2018 12:07:22 +0000 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <08a713f5-d1e0-4283-95ab-f59cff67185d@Spark> Message-ID: I'm assuming that it is expected behavior that the 21-rc1 tarball was removed from the downloads directory (The windows binaries are still there btw, as is the README). I found this because I ran `kerl update releases` and to my surprise it wasn't there today, whereas in the weekend it was. Is this an oversight, or should I await further development as things need to move around some more? On Mon, May 7, 2018 at 10:00 AM Lukas Larsson wrote: > Hello, > > On Fri, May 4, 2018 at 7:23 PM, Micha? Muska?a wrote: > >> The release looks great. Thank you to the whole OTP team for the work you >> put into it. >> > > Thanks! > > >> >> I wonder about the new file implementation that uses NIFs instead of >> ports, specifically the instrumentation around it. With ports, there was a >> way to list all ports and filter those that are files. This proved >> sometimes useful in debugging situations. Would it be possible to also get >> some kind of API to list all opened files, possibly with some additional >> information (like path)? >> I imagine a need for an API like that would be even more pressing once >> sockets are ported to use NIFs instead of ports as well. >> > > Yes, I also believe that something like this is going to be needed. While > it is possible on most OSs to look in the /proc/ fs to see which files are > opened, it is definitely useful to be able to track back to the process > controlling it. Even more so for sockets. > > Personally I would prefer not to have to build something new within erts > to handle this, but instead use what we have. Maybe ets tables that keeps > track of files, connections, etc. > > Each sockets will most likely have to be associated with a process, so for > them it could be possible to just store something in the process dictionary > and let the debug process get the pdict of all processes. > > Lukas > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Wed May 9 10:44:32 2018 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Wed, 9 May 2018 10:44:32 +0200 Subject: [erlang-questions] Patch package OTP 19.3.6.9 released Message-ID: Patch Package: OTP 19.3.6.9 Git Tag: OTP-19.3.6.9 Date: 2018-05-09 Trouble Report Id: OTP-15018, OTP-15064, OTP-15066 Seq num: System: OTP Release: 19 Application: ssh-4.4.2.4 Predecessor: OTP 19.3.6.8 Check out the git tag OTP-19.3.6.9, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- ssh-4.4.2.4 ----------------------------------------------------- --------------------------------------------------------------------- Note! The ssh-4.4.2.4 application can *not* be applied independently of other applications on an arbitrary OTP 19 installation. On a full OTP 19 installation, also the following runtime dependencies have to be satisfied: -- crypto-3.7.3 (first satisfied in OTP 19.3) -- public_key-1.4 (first satisfied in OTP 19.3) -- stdlib-3.3 (first satisfied in OTP 19.3) --- Fixed Bugs and Malfunctions --- OTP-15018 Application(s): ssh Fix rare spurios shutdowns of ssh servers when receiveing {'EXIT',_,normal} messages. OTP-15064 Application(s): ssh Host key hash erroneously calculated for clients following draft-00 of RFC 4419, for example PuTTY OTP-15066 Application(s): ssh Renegotiation could fail in some states Full runtime dependencies of ssh-4.4.2.4: crypto-3.7.3, erts-6.0, kernel-3.0, public_key-1.4, stdlib-3.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From getonga@REDACTED Wed May 9 11:40:51 2018 From: getonga@REDACTED (=?utf-8?B?Z2V0b25nYQ==?=) Date: Wed, 9 May 2018 17:40:51 +0800 Subject: [erlang-questions] gen_statem badrecord,trans_opts Message-ID: my own gen_statem module crash with this error msg:``` Context: child_terminated Reason: {{badrecord,trans_opts},[{gen_statem,loop_event_actions_list,10,[{file,"gen_statem.erl"},{line,1210}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} ``` I recheck my code many times, and not found any error in my return state. And the module runs for a long time, a week or some more, and this error happens for only two times. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Wed May 9 12:14:59 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 9 May 2018 12:14:59 +0200 Subject: [erlang-questions] gen_statem badrecord,trans_opts In-Reply-To: References: Message-ID: <20180509101459.GA98890@erix.ericsson.se> On Wed, May 09, 2018 at 05:40:51PM +0800, getonga wrote: > my own gen_statem module crash with this error msg:``` > Context: child_terminated > Reason: {{badrecord,trans_opts},[{gen_statem,loop_event_actions_list,10,[{file,"gen_statem.erl"},{line,1210}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} > > ``` > I recheck my code many times, and not found any error in my return state. And the module runs for a long time, a week or some more, and this error happens for only two times. Which Erlang/OTP version or which stdlib version? -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From raimo+erlang-questions@REDACTED Wed May 9 12:25:45 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 9 May 2018 12:25:45 +0200 Subject: [erlang-questions] gen_statem badrecord,trans_opts In-Reply-To: <20180509101459.GA98890@erix.ericsson.se> References: <20180509101459.GA98890@erix.ericsson.se> Message-ID: <20180509102545.GB98890@erix.ericsson.se> On Wed, May 09, 2018 at 12:14:59PM +0200, Raimo Niskanen wrote: > On Wed, May 09, 2018 at 05:40:51PM +0800, getonga wrote: > > my own gen_statem module crash with this error msg:``` > > Context: child_terminated > > Reason: {{badrecord,trans_opts},[{gen_statem,loop_event_actions_list,10,[{file,"gen_statem.erl"},{line,1210}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} > > > > ``` > > I recheck my code many times, and not found any error in my return state. And the module runs for a long time, a week or some more, and this error happens for only two times. > > Which Erlang/OTP version or which stdlib version? From the line number it appears to be Erlang/OTP-20.3.2 or later but before 21.0-rc1. It looks like a bug I fixed in commit 3ed7d729cab697b9f668dadb563d629de10f593d merged into master and included in 21.0-rc1. Can you try that version? -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From codewiget95@REDACTED Wed May 9 17:03:20 2018 From: codewiget95@REDACTED (asdf asdf) Date: Wed, 9 May 2018 11:03:20 -0400 Subject: [erlang-questions] Send IP Message with Custom IP Header Message-ID: Hi, I am using pkt to create a custom ipv4 header: ?IpHeader_ = #ipv4{p = 1, saddr = {127,0,0,1}, daddr = {192,168,7,83}, len = byte_size(Icmp) + 20}, ?IpHeader = pkt:ipv4(IpHeader_#ipv4{sum = ?pkt:makesum(IpHeader_)}) Using a static icmp echo packet: ?Packet = <<8,0,140,250,29,138,0,0,255,253,243,182,73,166,226,218,32,33,34,35,36,37, ? ? ? ?38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61, ? ? ? ?62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79>>, I can send the ICMP properly and see it through Wireshark using the following: ? ? {ok, FD} = procket:socket(inet, raw, icmp), ? ? {ok, Socket} = gen_udp:open(0, [binary, {fd, FD}, inet]), ?gen_udp:send(Socket, {192,168,7,83}, 0, Packet), Now, if I Include the icmp header: NewPacket = <> and send this, Wireshark shows a malformed ICMP packet. Gen_udp or procket, or someone down the line is still placing a standard icmp header on top of my custom one and treating it as payload. I have tried setting raw C setsocketopts : inet:sockopts(Socket, [{raw, 0(IPPROTO_IP), 3 (IP_HDRINCL) , <<1:32/native>>(integer 1 for on)}]) to attempt to stop the socket from adding the header, but when I do this the packet doesn?t seem to leave the machine, and I can?t see it on Wireshark. The numbers for IPPROTO comes from the header file:?https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/in.h?and IP_HDRINCL:?http://students.mimuw.edu.pl/SO/Linux/Kod/include/linux/socket.h.html?. Is there any way to put a custom IP header on any ip packet in erlang? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From zzantozz@REDACTED Wed May 9 17:30:35 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Wed, 09 May 2018 15:30:35 +0000 Subject: [erlang-questions] New SSL option to set TLS record version? Message-ID: I've run across a faulty SSL server implementation that appears to send a "handshake failure" alert if the ClientHello protocol version isn't equal to the TLS record version. In Erlang, different major versions choose the TLS record version differently. None of them are wrong according the TLS spec, but some of them break when I'm trying to connect to these bad server implementations. What do you think of adding a new ssl_option like "client_hello_tls_record_version" to let us explicitly set the version to be used? Ideally, it would support values like 'tlsv1', 'tlsv1_2', 'lowest', 'highest', and 'same_as_client_hello', for example. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed May 9 18:54:31 2018 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 9 May 2018 17:54:31 +0100 Subject: [erlang-questions] Disabling IPv6 name lookups? Message-ID: tl;dr: how can I prevent amqp_client from using inet6? We recently ran into a problem in our integration tests where connecting to a RabbitMQ server on localhost was taking too long, causing the tests to time out. It turns out that we were bitten by a set of overlapping issues: 1. amqp_connection:start(#amqp_params_network{host = "localhost"}) attempts to resolve the name "localhost" as both IPv6 and IPv4. 2. On Ubuntu, "localhost" doesn't have an IPv4 address (you're supposed to use "ip6-localhost"). 3. We had a couple of custom search suffixes in /etc/resolv.conf. 4. One of those suffixes was configured (in dnsmasq) to forward to another resolver that didn't exist. This resulted in ~8 seconds for each AMQP connection, and a timeout for the tests. It seems that it's possible to configure amqp_client to resolve the names using IPv4 and then IPv6, but I don't see that that's going to make a lot of difference, since it'll still try to resolve both. It doesn't seem to be possible to disable 'inet6' in amqp_client. So, my question: since we're not using IPv6, is there any way to tell Erlang (OTP-20.3.1) to globally disable the inet6 address family, and to just insta-fail any DNS queries that use it? Regards, Roger. From getonga@REDACTED Thu May 10 03:18:29 2018 From: getonga@REDACTED (=?utf-8?B?Z2V0b25nYQ==?=) Date: Thu, 10 May 2018 09:18:29 +0800 Subject: [erlang-questions] gen_statem badrecord,trans_opts Message-ID: my own gen_statem module crash with this error msg:``` Context: child_terminated Reason: {{badrecord,trans_opts},[{gen_statem,loop_event_actions_list,10,[{file,"gen_statem.erl"},{line,1210}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} ``` I recheck my code many times, and not found any error in my return state. And the module runs for a long time, a week or some more, and this error happens for only two times.My erlang version is 20.3.6, and my production environment is that, could you merge 3ed7d729cab697b9f668dadb563d629de10f593d into branch 20?erlang 21 is now rc1, I can not put 21 into production now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From karlsson.rm@REDACTED Fri May 11 15:15:08 2018 From: karlsson.rm@REDACTED (Mikael Karlsson) Date: Fri, 11 May 2018 15:15:08 +0200 Subject: [erlang-questions] Erlang on RISC-V In-Reply-To: References: <20180220104958.GA7895@hec.corelatus.se> Message-ID: 2018-02-20 14:45 GMT+01:00 Mikael Pettersson : > On Tue, Feb 20, 2018 at 11:49 AM, Matthias Lang > wrote: > > On 19. February 2018, Mikael Karlsson wrote: > > > >> has anyone looked at having the Erlang compiler support RISC-V, > > > > I haven't. > > > > What do you mean by "the Erlang compiler"? > > > > The regular Erlang compiler, i.e. the one that generates .beam files, > > generates .beam files which don't care what CPU you want to run on. > > > > HiPE is a different story, but you didn't mention HiPE. > > RISC-V support in HiPE should be straight-forward, but no-one is > working on that AFAIK. > It's not clear there is any need, since the BEAM should work as-is. > > There is also that HiPE/LLVM thing which might be able to support RISC-V > with less effort. > > > Getting the VM to run is another task, though once people have linux > > and a toolchain running (I see that there's an active debian effort, > > for instance), I would expect that to be the same level of difficulty > > as compiling on, say, MIPS linux for the first time---i.e. maybe it'll > > just work, maybe you have to do a bit of autotools fiddling. > > RISC-V has full GNU/Linux support with latest GCC, binutils, GLIBC, > and Linux kernel, and I know both Debian and Fedora have started ports. > But you don't need to wait for a full distro port, all you need is a > cross-compiler > and busybox or sth and you're done. > > > The mention of RISC-V just made me think, "I haven't heard anything > > about Tilera for a while". I think it's been a few years since anyone > > here said anything about it on the mailing list. Anyone know if it's > > still alive? > > Tilera is dropping their Tile processors and migrating to some other core > (I don't know which one, but probably ARM or MIPS.) See e.g. their recent > announcements to drop the older TilePro from Linux and glibc, keeping only > Tile-Gx due to some customers still using it. > Hi again Mikael, >It's not clear there is any need, since the BEAM should work as-is. Yes, the Fedora port you pointed out has already an Erlang port. However the RISC-V is a register machine (as the Erlang VM) with 32 integer registers so I wonder it there are any optimizations that may be done. I can see in the beam_emu.c a can see for other architectures sections like: /* * On certain platforms, make sure that the main variables really are placed * in registers. */ .. #elif defined(__GNUC__) && defined(__amd64__) && !defined(DEBUG) # define REG_xregs asm("%r12") # define REG_htop # define REG_stop asm("%r13") # define REG_I asm("%rbx") # define REG_fcalls asm("%r14") #else .. The RISC-V assembly programmer's handbook lists registers as (chapter 20 in the spec): Register ABI Name Description Saver x0 zero Hard-wired zero | x1 ra Return address Caller x2 sp Stack pointer Callee x3 gp Global pointer | x4 tp Thread pointer | x5 t0 Temporary/alternate link register Caller x6-7 t1-2 Temporaries Caller x8 s0/fp Saved register/frame pointer Callee x9 s1 Saved register Callee x10-11 a0-1 Function arguments/return values Caller x12-17 a2-7 Function arguments Caller x18-27 s2-11 Saved registers Callee x28-31 t3-6 Temporaries Caller So I guess some similar "reservations" can be made for RISC-V as for amd64, but I am not sure how to map them here. Best Regards Mikael K -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Fri May 11 18:52:50 2018 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Fri, 11 May 2018 18:52:50 +0200 Subject: [erlang-questions] Erlang on RISC-V In-Reply-To: References: <20180220104958.GA7895@hec.corelatus.se> Message-ID: On Fri, May 11, 2018 at 3:15 PM, Mikael Karlsson wrote: >>It's not clear there is any need, since the BEAM should work as-is. > > Yes, the Fedora port you pointed out has already an Erlang port. > > However the RISC-V is a register machine (as the Erlang VM) with 32 integer > registers so I wonder it there are any optimizations that may be done. > I can see in the beam_emu.c a can see for other architectures sections like: > /* > * On certain platforms, make sure that the main variables really are placed > * in registers. > */ > .. > #elif defined(__GNUC__) && defined(__amd64__) && !defined(DEBUG) > # define REG_xregs asm("%r12") > # define REG_htop > # define REG_stop asm("%r13") > # define REG_I asm("%rbx") > # define REG_fcalls asm("%r14") > #else > .. > The RISC-V assembly programmer's handbook lists registers as (chapter 20 in > the spec): > Register ABI Name Description Saver > x0 zero Hard-wired zero | > x1 ra Return address Caller > x2 sp Stack pointer Callee > x3 gp Global pointer | > x4 tp Thread pointer | > x5 t0 Temporary/alternate link register Caller > x6-7 t1-2 Temporaries Caller > x8 s0/fp Saved register/frame pointer Callee > x9 s1 Saved register Callee > x10-11 a0-1 Function arguments/return values Caller > x12-17 a2-7 Function arguments Caller > x18-27 s2-11 Saved registers Callee > x28-31 t3-6 Temporaries Caller > > So I guess some similar "reservations" can be made for RISC-V as for amd64, > but I am not sure how to map them here. This can be done of course. The code reserves callee-save registers for important BEAM VM variables on SPARC and AMD64, so you should identify 5 callee-save registers in the RISC-V ABI and bind them in a similar way. Your list is a bit unclear, but I guess the s0-s11 registers are callee-save; pick say the last 5. It's probably not worth trying to bind even more BEAM VM variables to registers, as that means more work in transitions in to and out of process_main(). You'll want to run the estone SUITE to see how much of a difference register variables make on RISC-V. /Mikael From karlsson.rm@REDACTED Fri May 11 19:30:22 2018 From: karlsson.rm@REDACTED (Mikael Karlsson) Date: Fri, 11 May 2018 19:30:22 +0200 Subject: [erlang-questions] Erlang on RISC-V In-Reply-To: References: <20180220104958.GA7895@hec.corelatus.se> Message-ID: 2018-05-11 18:52 GMT+02:00 Mikael Pettersson : > > On Fri, May 11, 2018 at 3:15 PM, Mikael Karlsson wrote: > >>It's not clear there is any need, since the BEAM should work as-is. > > > > Yes, the Fedora port you pointed out has already an Erlang port. > > > > However the RISC-V is a register machine (as the Erlang VM) with 32 integer > > registers so I wonder it there are any optimizations that may be done. > ... > > > So I guess some similar "reservations" can be made for RISC-V as for amd64, > > but I am not sure how to map them here. > > This can be done of course. The code reserves callee-save registers for important > BEAM VM variables on SPARC and AMD64, so you should identify 5 callee-save > registers in the RISC-V ABI and bind them in a similar way. Your list is a bit unclear, > but I guess the s0-s11 registers are callee-save; pick say the last 5. > > It's probably not worth trying to bind even more BEAM VM variables to > registers, as that means more work in transitions in to and out of process_main(). > > You'll want to run the estone SUITE to see how much of a difference > register variables make on RISC-V. Great, many thanks for the info. Also for anyone interested the last RISC-V workshop just finished in Barcelona, https://riscv.org/2018/04/risc-v-workshop-in-barcelona-agenda/ /Mikael K From wadt18@REDACTED Fri May 11 21:58:27 2018 From: wadt18@REDACTED (WADT 2018) Date: Fri, 11 May 2018 19:58:27 +0000 Subject: [erlang-questions] WADT 2018 & Leverhulme School - New extension to abstract submission deadline! Message-ID: ====================================================================== LAST CALL FOR PAPERS - New extension to abstract submission deadline! WADT 2018 24th International Workshop on Algebraic Development Techniques http://wadt18.cs.rhul.ac.uk July 2?5, 2018, Royal Holloway University of London, Egham, UK AND CALL FOR PARTICIPATION - Programme now available! Leverhulme School on Graph Transformation Techniques 2018 http://wadt18.cs.rhul.ac.uk/lsgt June 29?July 1, 2018, Royal Holloway University of London, Egham, UK ====================================================================== AIMS AND SCOPE The algebraic approach to system specification encompasses many aspects of the formal design of software systems. Originally born as a formal method for reasoning about abstract data types, it now covers new specification frameworks and programming paradigms (such as object-oriented, aspect-oriented, agent-oriented, logic and higher-order functional programming) as well as a wide range of application areas (including information systems, concurrent, distributed and mobile systems). The workshop will provide an opportunity to present recent and ongoing work, to meet colleagues, and to discuss new ideas and future trends. TOPICS OF INTEREST Typical, but not exclusive topics of interest are: ? Foundations of algebraic specification ? Other approaches to formal specification, including process calculi and models of concurrent, distributed, and cyber-physical systems ? Specification languages, methods, and environments ? Semantics of conceptual modelling methods and techniques ? Model-driven development ? Graph transformations, term rewriting, and proof systems ? Integration of formal specification techniques ? Formal testing and quality assurance, validation, and verification ? Algebraic approaches to cognitive sciences, including computational creativity WORKSHOP FORMAT AND LOCATION The workshop will take place over four days, Monday to Thursday, at Royal Holloway University of London in Egham, UK (https://www.royalholloway.ac.uk). Presentations will be selected on the basis of submitted abstracts. Leverhulme School on Graph Transformation Techniques This occurrence of the ADT workshop will be preceded by the Leverhulme School on Graph Transformation Techniques. The school will take place over three days, from Friday, June 29th, to Sunday, July 1st, and will comprise a self-contained series of invited lectures on graph transformation to be given by Reiko Heckel (University of Leicester, UK), Fernando Orejas (Technical University of Catalonia, Spain), and Detlef Plump (University of York, UK). INVITED SPEAKERS Artur d'Avila Garcez (City, University of London, UK) Rolf Hennicker (LMU Munich, Germany) Kai-Uwe K?hnberger (Osnabr?ck University, Germany) Fernando Orejas (Technical University of Catalonia, Spain) IMPORTANT DATES Submission deadline for abstracts: May 18th, 2018 Notification of acceptance: May 25th, 2018 Early registration: June 1st, 2018 Final abstract due: June 8th, 2018 Leverhulme School on Graph Transformation Techniques: June 29?July 1, 2018 ADT Workshop: July 2?5, 2018 SUBMISSIONS The scientific programme of the workshop will include presentations of recent results or ongoing research as well as invited talks. The presentations will be selected by the Steering Committee on the basis of submitted abstracts according to originality, significance and general interest. Abstracts must not exceed two pages including references; if a longer version of the contribution is available, it can be made accessible on the web and referenced in the abstract. Abstracts have to be submitted electronically via the EasyChair system at https://easychair.org/conferences/?conf=wadt18. PROCEEDINGS After the workshop, authors will be invited to submit full papers for the refereed proceedings. All submissions will be reviewed by the Programme Committee. Selection will be based on originality, soundness, and significance of the presented ideas and results. The proceedings will be published as a volume of Lecture Notes in Computer Science (Springer). The deadline for submissions will be September 3, 2018, with notifications by October 29. Camera-ready versions will be required by November 11. SPONSORSHIP The workshop takes place under the auspices of IFIP WG 1.3, while the series of graph-transformation lectures is supported by a Leverhulme Trust Visiting Professorship. WADT STEERING COMMITTEE Andrea Corradini (Italy) Jos? Fiadeiro (UK) [co-chair] Rolf Hennicker (Germany) Hans-J?rg Kreowski (Germany) Till Mossakowski (Germany) Fernando Orejas (Spain) Markus Roggenbach (UK) Grigore Ro?u (United States) PROGRAMME COMMITEE Paolo Baldan (Italy) Andrea Corradini (Italy) Artur d'Avila Garcez (UK) R?zvan Diaconescu (Romania) Jos? Fiadeiro (UK) [co-chair] Fabio Gadducci (Italy) Reiko Heckel (UK) Rolf Hennicker (Germany) Alexander Knapp (Germany) Barbara K?nig (Germany) Ant?nia Lopes (Portugal) Narciso Marti-Oliet (Spain) Till Mossakowski (Germany) Fernando Orejas (Spain) Leila Ribeiro (Brazil) Markus Roggenbach (UK) Pierre-Yves Schobbens (Belgium) Lutz Schr?der (Germany) Pawel Sobocinski (UK) Ionu? ?u?u (UK) [co-chair] Martin Wirsing (Germany) ORGANIZING COMMITTEE Claudia Chiri?? (UK) Jos? Fiadeiro (UK) Ionu? ?u?u (UK) CONTACT INFORMATION Email: wadt18@REDACTED Homepage: http://wadt18.cs.rhul.ac.uk From raimo+erlang-questions@REDACTED Mon May 14 10:47:30 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 14 May 2018 10:47:30 +0200 Subject: [erlang-questions] gen_statem badrecord,trans_opts In-Reply-To: References: Message-ID: <20180514084730.GA48381@erix.ericsson.se> On Thu, May 10, 2018 at 09:18:29AM +0800, getonga wrote: > my own gen_statem module crash with this error msg: > ``` > Context: child_terminated Reason: > {{badrecord,trans_opts}, > [{gen_statem,loop_event_actions_list,10, > [{file,"gen_statem.erl"},{line,1210}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,247}]}]} > ``` > I recheck my code many times, and not found any error in my return state. > And the module runs for a long time, a week or some more, and this error happens > for only two times. My erlang version is 20.3.6, and my production environment > is that, could you merge 3ed7d729cab697b9f668dadb563d629de10f593d > into branch 20?erlang 21 is now rc1, I can not put 21 into production now. What I can read and remember from commit 3ed7d72 is that when gen_statem is about to terminate due to a parse error of the Actions list it instead crashes because the TransOpts variable is in this case not a record. The commit fixes the crash to become a proper server termination. But the root cause should be a bad Action in the Actions list. The rest of the commit tightens parsing of timeout actions so that some erroneous actions that previously slipped through now fails the argument parsing. In particular the first element of a timeout tuple: {TimeoutType,Time,Msg} | {TimeoutType,Time,Msg,Opts} Here TimeoutType may be 'timeout' | 'state_timeout' | {'timeout',_} but that was not checked before this commit so misspelled timers worked, hence {AnyTerm,Time,Msg} would before this commit start a timeout. So I would think there is some erroneous timeout action returned from your callback module, and that merging this commit into the 20 track will not really help, just improve the server termination and diagnostics... Can you apply that commit yourself and tell me what happens? Best Regards -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From per@REDACTED Mon May 14 19:54:00 2018 From: per@REDACTED (Per Hedeland) Date: Mon, 14 May 2018 19:54:00 +0200 Subject: [erlang-questions] Handshake failure with DHE-DSS cipher suites Message-ID: <3e7e998e-a807-bb1a-dc0e-3ebbbe753fcd@hedeland.org> Hi, For no other reason than the fact that our documentation claims that we support all the ciphers listed by ssl:cipher_suites(openssl) (users can select a subset by configuring a list of OpenSSL names), I actually have a test that tries them individually, and in OTP 20 ssl:ssl_accept() fails with {error, {tls_alert,"handshake failure"}} for all the DHE-DSS suites - i.e. DHE-DSS-AES256-SHA256 DHE-DSS-AES128-SHA256 DHE-DSS-AES256-SHA DHE-DSS-AES128-SHA DHE-DSS-AES256-GCM-SHA384 DHE-DSS-AES128-GCM-SHA256 a.k.a. {dhe_dss,aes_256_cbc,sha256} {dhe_dss,aes_128_cbc,sha256} {dhe_dss,aes_256_cbc,sha} {dhe_dss,aes_128_cbc,sha} {dhe_dss,aes_256_gcm,aead,sha384} {dhe_dss,aes_128_gcm,aead,sha256} After debugging a bit with the first one, I found that the failure was due to a function_clause error, and that this change made it work: --- a/lib/public_key/src/public_key.erl +++ b/lib/public_key/src/public_key.erl @@ -505,7 +505,9 @@ pkix_sign_types(?'ecdsa-with-SHA256') -> pkix_sign_types(?'ecdsa-with-SHA384') -> {sha384, ecdsa}; pkix_sign_types(?'ecdsa-with-SHA512') -> - {sha512, ecdsa}. + {sha512, ecdsa}; +pkix_sign_types({2,16,840,1,101,3,4,3,2}) -> + {sha256, dsa}. %%-------------------------------------------------------------------- -spec sign(binary() | {digest, binary()}, Some googling indicates that the oid-tuple could reasonably be named 'id-dsa-with-sha256', no real surprise there - a bit more surprising is that the change actually makes *all* of those suites work. Now, I don't really know what I'm doing here, so: Is this just an omission in the public_key code and the above a reasonable fix, or is there some fundamental reason that pkix_sign_types/1 shouldn't handle that oid? (And if the latter, how can the DHE-DSS suites be made to work without it?) The test was run with 'openssl s_client' connecting to 20.3.6 - I can't easily plug 21.0-rc1 into the test, but it seems at least public_key:pkix_sign_types/1 is the same there. Thanks --Per Hedeland From zzantozz@REDACTED Mon May 14 21:55:44 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Mon, 14 May 2018 14:55:44 -0500 Subject: [erlang-questions] Global lock is unfair? Message-ID: My team has used global:set_lock() to implement a sort of simple mutex system on a set of resources. It turns out that this function seems to have no concept of fairness. One process might request a lock and not get it until 20 other processes have requested and gotten said lock, out of request order. Is there any easy way to do fair locking? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Mon May 14 22:41:26 2018 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 14 May 2018 13:41:26 -0700 Subject: [erlang-questions] Global lock is unfair? In-Reply-To: References: Message-ID: <843fe2aa-4533-bf8b-fd15-c258ec9b8ae3@gmail.com> Use a single Erlang process.? It will only be able to handle a single message at a time (other messages will get queued in-order) and the behavior will be fair. On 05/14/2018 12:55 PM, Ryan Stewart wrote: > My team has used global:set_lock() to implement a sort of simple mutex system on a set of resources. It turns out that this function seems to have no concept of fairness. One process might request a lock and not get it until 20 other processes have requested and gotten said lock, out of request order. Is there any easy way to do fair locking? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From zambal@REDACTED Mon May 14 22:57:18 2018 From: zambal@REDACTED (Vincent Siliakus) Date: Mon, 14 May 2018 22:57:18 +0200 Subject: [erlang-questions] Issue with using enif_binary_to_term Message-ID: Hi all, I'm writing a NIF library and can't wrap my head around why the following code makes the erlang runtime hang when called from a shell: static ERL_NIF_TERM test(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) { ErlNifBinary bin; ERL_NIF_TERM list = enif_make_list(env, 0); ERL_NIF_TERM in_term = enif_make_uint(env, 42); ERL_NIF_TERM out_term1, out_term2, out_term3; enif_term_to_binary(env, in_term, &bin); enif_binary_to_term(env, bin.data, bin.size, &out_term1, 0); list = enif_make_list_cell(env, out_term1, list); enif_binary_to_term(env, bin.data, bin.size, &out_term2, 0); list = enif_make_list_cell(env, out_term2, list); enif_binary_to_term(env, bin.data, bin.size, &out_term3, 0); list = enif_make_list_cell(env, out_term3, list); return list; } The multiple calls to enif_binary_to_term somehow seem to corrupt memory in the calling environment, so I'm probably using it incorrectly. Could some kind soul point me to the error? I'm running this code on OTP20 / erts-9.2. Thanks in advance, Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: From zzantozz@REDACTED Mon May 14 23:53:08 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Mon, 14 May 2018 16:53:08 -0500 Subject: [erlang-questions] Global lock is unfair? In-Reply-To: <843fe2aa-4533-bf8b-fd15-c258ec9b8ae3@gmail.com> References: <843fe2aa-4533-bf8b-fd15-c258ec9b8ae3@gmail.com> Message-ID: Funneling all the calls through a single process would bring the system to a grinding halt. Sure, I could write a system of processes, each of which acts as a mutex to one, particular resource, but why would I want to? All I need is locking, just like global locks, but fair. It seems like something that should already exist in a proven library. On Mon, May 14, 2018 at 3:41 PM Michael Truog wrote: > Use a single Erlang process. It will only be able to handle a single > message at a time (other messages will get queued in-order) and the > behavior will be fair. > > On 05/14/2018 12:55 PM, Ryan Stewart wrote: > > My team has used global:set_lock() to implement a sort of simple mutex > system on a set of resources. It turns out that this function seems to have > no concept of fairness. One process might request a lock and not get it > until 20 other processes have requested and gotten said lock, out of > request order. Is there any easy way to do fair locking? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From getonga@REDACTED Tue May 15 08:23:08 2018 From: getonga@REDACTED (=?utf-8?B?Z2V0b25nYQ==?=) Date: Tue, 15 May 2018 14:23:08 +0800 Subject: [erlang-questions] gen_statem badrecord,trans_opts Message-ID: Thanks, I recheck my source code, and I found the error reason. Thanks for your time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangud@REDACTED Tue May 15 09:46:43 2018 From: dangud@REDACTED (Dan Gudmundsson) Date: Tue, 15 May 2018 09:46:43 +0200 Subject: [erlang-questions] Global lock is unfair? In-Reply-To: References: <843fe2aa-4533-bf8b-fd15-c258ec9b8ae3@gmail.com> Message-ID: If you are already using mnesia you can use mnesia:lock({global, GlobalKey, Nodes}, write|read) but you will have to run it in a transaction on nodes where mnesia is running. /Dan On Mon, May 14, 2018 at 11:53 PM Ryan Stewart wrote: > Funneling all the calls through a single process would bring the system to > a grinding halt. Sure, I could write a system of processes, each of which > acts as a mutex to one, particular resource, but why would I want to? All I > need is locking, just like global locks, but fair. It seems like something > that should already exist in a proven library. > > > On Mon, May 14, 2018 at 3:41 PM Michael Truog wrote: > >> Use a single Erlang process. It will only be able to handle a single >> message at a time (other messages will get queued in-order) and the >> behavior will be fair. >> >> On 05/14/2018 12:55 PM, Ryan Stewart wrote: >> >> My team has used global:set_lock() to implement a sort of simple mutex >> system on a set of resources. It turns out that this function seems to have >> no concept of fairness. One process might request a lock and not get it >> until 20 other processes have requested and gotten said lock, out of >> request order. Is there any easy way to do fair locking? >> >> _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue May 15 14:39:36 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 15 May 2018 15:39:36 +0300 Subject: [erlang-questions] how to start and stop erlang daemon Message-ID: Hi. I've written a bit unstructured thoughts about seamless start and stop of erlang daemon: https://medium.com/@max.lapshin/how-to-start-and-stop-erlang-daemon-3fd988777ab3 In short: 1) validate config before launching 2) control that daemon has started 3) find a way to notify why it hasn't started 4) use several ways to stop it and to ensure that it has stopped -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Tue May 15 15:08:16 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Tue, 15 May 2018 15:08:16 +0200 Subject: [erlang-questions] Handshake failure with DHE-DSS cipher suites In-Reply-To: <3e7e998e-a807-bb1a-dc0e-3ebbbe753fcd@hedeland.org> References: <3e7e998e-a807-bb1a-dc0e-3ebbbe753fcd@hedeland.org> Message-ID: Hi! I think that the oid is missing due to an oversight in adding it when crypto was updated to support it. For some reason (which I do not know) early versions of crypto did only support sha1 (sha) with dsa. The correct oid name should be in the generated include file included in public_key.hrl Without looking into this further, I suspect that the passing of the suites, could be due to that the actual sha function used is a result of the "hash_sign negotiation" that depends on the hash_sign hello extension. Regards Ingela Erlang/OTP team - Ericsson AB 2018-05-14 19:54 GMT+02:00 Per Hedeland : > Hi, > > For no other reason than the fact that our documentation claims that we > support all the ciphers listed by ssl:cipher_suites(openssl) (users can > select a subset by configuring a list of OpenSSL names), I actually have > a test that tries them individually, and in OTP 20 ssl:ssl_accept() > fails with {error, {tls_alert,"handshake failure"}} for all the DHE-DSS > suites - i.e. > > DHE-DSS-AES256-SHA256 > DHE-DSS-AES128-SHA256 > DHE-DSS-AES256-SHA > DHE-DSS-AES128-SHA > DHE-DSS-AES256-GCM-SHA384 > DHE-DSS-AES128-GCM-SHA256 > > a.k.a. > > {dhe_dss,aes_256_cbc,sha256} > {dhe_dss,aes_128_cbc,sha256} > {dhe_dss,aes_256_cbc,sha} > {dhe_dss,aes_128_cbc,sha} > {dhe_dss,aes_256_gcm,aead,sha384} > {dhe_dss,aes_128_gcm,aead,sha256} > > After debugging a bit with the first one, I found that the failure was > due to a function_clause error, and that this change made it work: > > --- a/lib/public_key/src/public_key.erl > +++ b/lib/public_key/src/public_key.erl > @@ -505,7 +505,9 @@ pkix_sign_types(?'ecdsa-with-SHA256') -> > pkix_sign_types(?'ecdsa-with-SHA384') -> > {sha384, ecdsa}; > pkix_sign_types(?'ecdsa-with-SHA512') -> > - {sha512, ecdsa}. > + {sha512, ecdsa}; > +pkix_sign_types({2,16,840,1,101,3,4,3,2}) -> > + {sha256, dsa}. > > %%-------------------------------------------------------------------- > -spec sign(binary() | {digest, binary()}, > > Some googling indicates that the oid-tuple could reasonably be named > 'id-dsa-with-sha256', no real surprise there - a bit more surprising is > that the change actually makes *all* of those suites work. Now, I don't > really know what I'm doing here, so: Is this just an omission in the > public_key code and the above a reasonable fix, or is there some > fundamental reason that pkix_sign_types/1 shouldn't handle that oid? > (And if the latter, how can the DHE-DSS suites be made to work without > it?) > > The test was run with 'openssl s_client' connecting to 20.3.6 - I can't > easily plug 21.0-rc1 into the test, but it seems at least > public_key:pkix_sign_types/1 is the same there. > > Thanks > > --Per Hedeland > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Tue May 15 16:14:23 2018 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Tue, 15 May 2018 14:14:23 +0000 Subject: [erlang-questions] Issue with using enif_binary_to_term In-Reply-To: References: Message-ID: <1526393663.8310.6.camel@ericsson.com> This is a bug in enif_binary_to_term which causes heap corruption when the term is an immediate (atom, small integer, pid, port, empty list). This should fix it: diff --git a/erts/emulator/beam/erl_nif.c b/erts/emulator/beam/erl_nif.c index e208792..0fbf0eb 100644 --- a/erts/emulator/beam/erl_nif.c +++ b/erts/emulator/beam/erl_nif.c @@ -1255,8 +1255,10 @@ size_t enif_binary_to_term(ErlNifEnv *dst_env, ?????if (is_non_value(*term)) { ?????????return 0; ?????} -????erts_factory_close(&factory); -????cache_env(dst_env); +????if (size > 0) { +????????erts_factory_close(&factory); +????????cache_env(dst_env); +????} ? ?????ASSERT(bp > data); ?????return bp - data; Your usage looks correct. The only nitpick is to test the return value from enif_binary_to_term, either to handle broken binary or assert it's correct. /Sverker On m?n, 2018-05-14 at 22:57 +0200, Vincent Siliakus wrote: > Hi all, > > I'm writing a NIF library and can't wrap my head around why the following code > makes the erlang runtime hang when called from a shell: > > ? static ERL_NIF_TERM test(ErlNifEnv* env, int argc, const ERL_NIF_TERM > argv[]) { > ??? ErlNifBinary bin; > > ??? ERL_NIF_TERM list = enif_make_list(env, 0); > ??? ERL_NIF_TERM in_term = enif_make_uint(env, 42); > ??? ERL_NIF_TERM out_term1, out_term2, out_term3; > > ??? enif_term_to_binary(env, in_term, &bin); > > ??? enif_binary_to_term(env, bin.data, bin.size, &out_term1, 0); > ??? list = enif_make_list_cell(env, out_term1, list); > > ??? enif_binary_to_term(env, bin.data, bin.size, &out_term2, 0); > ??? list = enif_make_list_cell(env, out_term2, list); > > ??? enif_binary_to_term(env, bin.data, bin.size, &out_term3, 0); > ??? list = enif_make_list_cell(env, out_term3, list); > > ??? return list; > ? } > > The multiple calls to enif_binary_to_term somehow seem to corrupt memory in > the calling environment, so I'm probably using it incorrectly. Could some kind > soul point me to the error? I'm running this code on OTP20 / erts-9.2. > > Thanks in advance, > Vincent > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zambal@REDACTED Tue May 15 17:02:06 2018 From: zambal@REDACTED (Vincent Siliakus) Date: Tue, 15 May 2018 17:02:06 +0200 Subject: [erlang-questions] Issue with using enif_binary_to_term Message-ID: Hi Sverker, Thanks for the reply including the fix. Will there be an OTP 20 release that will include this patch, or will it only be included in OTP 21? For internal usage it doesn't really matter, but I'm hoping to release the library I'm working on as open source in the near future. Regarding testing the return value: I'm actually doing this in the code where I stumbled upon this issue, which is reading terms in external term format from an embedded kv store. I just wanted to make the example as simple as possible to highlight the issue. Best, Vincent > This is a bug in enif_binary_to_term which causes heap corruption when the term > is an immediate (atom, small integer, pid, port, empty list). > > This should fix it: > > diff --git a/erts/emulator/beam/erl_nif.c b/erts/emulator/beam/erl_nif.c > index e208792..0fbf0eb 100644 > --- a/erts/emulator/beam/erl_nif.c > +++ b/erts/emulator/beam/erl_nif.c > @@ -1255,8 +1255,10 @@ size_t enif_binary_to_term(ErlNifEnv *dst_env, > if (is_non_value(*term)) { > return 0; > } > - erts_factory_close(&factory); > - cache_env(dst_env); > + if (size > 0) { > + erts_factory_close(&factory); > + cache_env(dst_env); > + } > > ASSERT(bp > data); > return bp - data; > > > > Your usage looks correct. The only nitpick is to test the return value from > enif_binary_to_term, either to handle broken binary or assert it's correct. > > /Sverker > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zambal@REDACTED Tue May 15 17:15:18 2018 From: zambal@REDACTED (Vincent Siliakus) Date: Tue, 15 May 2018 17:15:18 +0200 Subject: [erlang-questions] Issue with using enif_binary_to_term In-Reply-To: References: Message-ID: Hi Ross, Wow, thanks for the extensive detective work! Once upon a time I was a full time C++ developer, only using Visual Studio and it's debugger. Gdb always looked so arcane in comparison, but this is an excellent reminder that there's no excuse for me to dig into gdb. In the mean time it's confirmed that my issue is caused by a bug in enif_binary_to_term, but many thanks for you time anyway. Best, Vincent On Tue, May 15, 2018 at 5:01 PM, Ross Schlaikjer wrote: > Vincent, > > I'm not entirely sure yet why this is happening, but I think I have at > least figured out at least some of what is going on. > > Heap allocation in erts is done very simply (for the most part), in that > it is linearly allocated like so (as of otp git tag OTP-19.2.1): > > 140 static ERTS_INLINE Eterm* alloc_heap(ErlNifEnv* env, size_t need) > 141 { > 142 Eterm* hp = env->hp; > 143 env->hp += need; > 144 if (env->hp <= env->hp_end) { > 145 return hp; > 146 } > 147 return alloc_heap_heavy(env, need, hp); > 148 } > > When allocating a new cons cell, we take two 64bit words, store car/cdr > and return the address of the base pointer + the list tag (0x1). > > 1726 ERL_NIF_TERM enif_make_list_cell(ErlNifEnv* env, Eterm car, Eterm cdr) > 1727 { > 1728 Eterm* hp = alloc_heap(env,2); > 1729 Eterm ret = make_list(hp); > 1730 > 1731 ASSERT_IN_ENV(env, car, 0, "head of list cell"); > 1732 ASSERT_IN_ENV(env, cdr, 0, "tail of list cell"); > 1733 CAR(hp) = car; > 1734 CDR(hp) = cdr; > 1735 return ret; > 1736 } > > make_list above epands to the macro > #define _unchecked_make_list(x) ((Uint)(x) + TAG_PRIMARY_LIST) > so all it's doing is adding 1 to that ret pointer. > > If we take this all to gdb, we can step through your code and see the list > modifications occurring (comments inline): > > 18 ERL_NIF_TERM list = enif_make_list(env, 0); > (gdb) n > 19 ERL_NIF_TERM in_term = enif_make_uint(env, 42); > (gdb) p/x list > $1 = 0xfffffffffffffffb # This is the 'empty list' value > (gdb) n > 22 int success = enif_term_to_binary(env, in_term, &bin); > (gdb) n > 24 int bcount = enif_binary_to_term(env, bin.data, 3, &out_term1, 0); > (gdb) p/x env->hp # Check the heap pointer before we call > enif_binary_to_term > $4 = 0x7fff79840fe0 > (gdb) n > 25 list = enif_make_list_cell(env, out_term1, list); > (gdb) p/x env->hp # Check again after - notice the hp hasn't changed > $5 = 0x7fff79840fe0 > (gdb) n # Make the cons cell > 27 bcount = enif_binary_to_term(env, bin.data, 3, &out_term2, 0); > (gdb) p/x list # As we can see, it has the address of the base pointer + > the list tag (1) > $6 = 0x7fff79840fe1 > (gdb) x/2g list-1 # If we examine it, we see it has the head (our > immediate int value) and the tail (empty list) > 0x7fff79840fe0: 0x00000000000002af 0xfffffffffffffffb > (gdb) p/x env->hp # Our heap pointer has gone up by 16 bytes, as expected > (2 64bit words) > $7 = 0x7fff79840ff0 > (gdb) n # Now we execute the second enif_binary_to_term call > 28 list = enif_make_list_cell(env, out_term2, list); > (gdb) p/x env->hp > $8 = 0x7fff79840fe0 # This is where our problem begins! binary_to_term > has _decremented_ the heap pointer! > (gdb) n # Create our second cons cell > 30 bcount = enif_binary_to_term(env, bin.data, 3, &out_term3, 0); > (gdb) p/x list # We look at the 'new' list pointer value... but it's the > same as before! > $9 = 0x7fff79840fe1 > (gdb) x/2g list -1 > 0x7fff79840fe0: 0x00000000000002af 0x00007fff79840fe1 # This list now > references itself! > > It looks like when we call binary_to_term, the cache_env call is what's > messing with our heap pointer - > (gdb) watch env->hp > Hardware watchpoint 2: env->hp > (gdb) n > > Thread 14 "1_scheduler" hit Hardware watchpoint 2: env->hp > > Old value = (Eterm *) 0x7fff79840ff0 > New value = (Eterm *) 0x7fff79840fe0 > 0x00005555556ce798 in cache_env (env=0x7fffb34bee40) at beam/erl_nif.c:421 > warning: Source file is more recent than executable. > 421 env->hp = HEAP_TOP(env->proc); > (gdb) bt > #0 0x00005555556ce798 in cache_env (env=0x7fffb34bee40) at > beam/erl_nif.c:421 > #1 enif_binary_to_term (dst_env=0x7fffb34bee40, data=0x7fffb5b82b68 > "\203a*", data_sz=, term=0x7fffb34bed28, > opts=(unknown: 0)) at beam/erl_nif.c:1161 > > > So as far as I can tell, something to do with the switchover between > HAlloc and alloc_heap (flush_env / cache_env) seems to be losing some bytes > off the heap pointer, but _only_ for subsequent calls. Unfortunately I must > go back to my day job for now, but will try and debug more later. > > Hope this helps. > > - Ross S > > On Mon, May 14, 2018 at 4:57 PM, Vincent Siliakus > wrote: > >> Hi all, >> >> I'm writing a NIF library and can't wrap my head around why the following >> code makes the erlang runtime hang when called from a shell: >> >> static ERL_NIF_TERM test(ErlNifEnv* env, int argc, const ERL_NIF_TERM >> argv[]) { >> ErlNifBinary bin; >> >> ERL_NIF_TERM list = enif_make_list(env, 0); >> ERL_NIF_TERM in_term = enif_make_uint(env, 42); >> ERL_NIF_TERM out_term1, out_term2, out_term3; >> >> enif_term_to_binary(env, in_term, &bin); >> >> enif_binary_to_term(env, bin.data, bin.size, &out_term1, 0); >> list = enif_make_list_cell(env, out_term1, list); >> >> enif_binary_to_term(env, bin.data, bin.size, &out_term2, 0); >> list = enif_make_list_cell(env, out_term2, list); >> >> enif_binary_to_term(env, bin.data, bin.size, &out_term3, 0); >> list = enif_make_list_cell(env, out_term3, list); >> >> return list; >> } >> >> The multiple calls to enif_binary_to_term somehow seem to corrupt memory >> in the calling environment, so I'm probably using it incorrectly. Could >> some kind soul point me to the error? I'm running this code on OTP20 / >> erts-9.2. >> >> Thanks in advance, >> Vincent >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rossschlaikjer@REDACTED Tue May 15 17:01:23 2018 From: rossschlaikjer@REDACTED (Ross Schlaikjer) Date: Tue, 15 May 2018 11:01:23 -0400 Subject: [erlang-questions] Issue with using enif_binary_to_term In-Reply-To: References: Message-ID: Vincent, I'm not entirely sure yet why this is happening, but I think I have at least figured out at least some of what is going on. Heap allocation in erts is done very simply (for the most part), in that it is linearly allocated like so (as of otp git tag OTP-19.2.1): 140 static ERTS_INLINE Eterm* alloc_heap(ErlNifEnv* env, size_t need) 141 { 142 Eterm* hp = env->hp; 143 env->hp += need; 144 if (env->hp <= env->hp_end) { 145 return hp; 146 } 147 return alloc_heap_heavy(env, need, hp); 148 } When allocating a new cons cell, we take two 64bit words, store car/cdr and return the address of the base pointer + the list tag (0x1). 1726 ERL_NIF_TERM enif_make_list_cell(ErlNifEnv* env, Eterm car, Eterm cdr) 1727 { 1728 Eterm* hp = alloc_heap(env,2); 1729 Eterm ret = make_list(hp); 1730 1731 ASSERT_IN_ENV(env, car, 0, "head of list cell"); 1732 ASSERT_IN_ENV(env, cdr, 0, "tail of list cell"); 1733 CAR(hp) = car; 1734 CDR(hp) = cdr; 1735 return ret; 1736 } make_list above epands to the macro #define _unchecked_make_list(x) ((Uint)(x) + TAG_PRIMARY_LIST) so all it's doing is adding 1 to that ret pointer. If we take this all to gdb, we can step through your code and see the list modifications occurring (comments inline): 18 ERL_NIF_TERM list = enif_make_list(env, 0); (gdb) n 19 ERL_NIF_TERM in_term = enif_make_uint(env, 42); (gdb) p/x list $1 = 0xfffffffffffffffb # This is the 'empty list' value (gdb) n 22 int success = enif_term_to_binary(env, in_term, &bin); (gdb) n 24 int bcount = enif_binary_to_term(env, bin.data, 3, &out_term1, 0); (gdb) p/x env->hp # Check the heap pointer before we call enif_binary_to_term $4 = 0x7fff79840fe0 (gdb) n 25 list = enif_make_list_cell(env, out_term1, list); (gdb) p/x env->hp # Check again after - notice the hp hasn't changed $5 = 0x7fff79840fe0 (gdb) n # Make the cons cell 27 bcount = enif_binary_to_term(env, bin.data, 3, &out_term2, 0); (gdb) p/x list # As we can see, it has the address of the base pointer + the list tag (1) $6 = 0x7fff79840fe1 (gdb) x/2g list-1 # If we examine it, we see it has the head (our immediate int value) and the tail (empty list) 0x7fff79840fe0: 0x00000000000002af 0xfffffffffffffffb (gdb) p/x env->hp # Our heap pointer has gone up by 16 bytes, as expected (2 64bit words) $7 = 0x7fff79840ff0 (gdb) n # Now we execute the second enif_binary_to_term call 28 list = enif_make_list_cell(env, out_term2, list); (gdb) p/x env->hp $8 = 0x7fff79840fe0 # This is where our problem begins! binary_to_term has _decremented_ the heap pointer! (gdb) n # Create our second cons cell 30 bcount = enif_binary_to_term(env, bin.data, 3, &out_term3, 0); (gdb) p/x list # We look at the 'new' list pointer value... but it's the same as before! $9 = 0x7fff79840fe1 (gdb) x/2g list -1 0x7fff79840fe0: 0x00000000000002af 0x00007fff79840fe1 # This list now references itself! It looks like when we call binary_to_term, the cache_env call is what's messing with our heap pointer - (gdb) watch env->hp Hardware watchpoint 2: env->hp (gdb) n Thread 14 "1_scheduler" hit Hardware watchpoint 2: env->hp Old value = (Eterm *) 0x7fff79840ff0 New value = (Eterm *) 0x7fff79840fe0 0x00005555556ce798 in cache_env (env=0x7fffb34bee40) at beam/erl_nif.c:421 warning: Source file is more recent than executable. 421 env->hp = HEAP_TOP(env->proc); (gdb) bt #0 0x00005555556ce798 in cache_env (env=0x7fffb34bee40) at beam/erl_nif.c:421 #1 enif_binary_to_term (dst_env=0x7fffb34bee40, data=0x7fffb5b82b68 "\203a*", data_sz=, term=0x7fffb34bed28, opts=(unknown: 0)) at beam/erl_nif.c:1161 So as far as I can tell, something to do with the switchover between HAlloc and alloc_heap (flush_env / cache_env) seems to be losing some bytes off the heap pointer, but _only_ for subsequent calls. Unfortunately I must go back to my day job for now, but will try and debug more later. Hope this helps. - Ross S On Mon, May 14, 2018 at 4:57 PM, Vincent Siliakus wrote: > Hi all, > > I'm writing a NIF library and can't wrap my head around why the following > code makes the erlang runtime hang when called from a shell: > > static ERL_NIF_TERM test(ErlNifEnv* env, int argc, const ERL_NIF_TERM > argv[]) { > ErlNifBinary bin; > > ERL_NIF_TERM list = enif_make_list(env, 0); > ERL_NIF_TERM in_term = enif_make_uint(env, 42); > ERL_NIF_TERM out_term1, out_term2, out_term3; > > enif_term_to_binary(env, in_term, &bin); > > enif_binary_to_term(env, bin.data, bin.size, &out_term1, 0); > list = enif_make_list_cell(env, out_term1, list); > > enif_binary_to_term(env, bin.data, bin.size, &out_term2, 0); > list = enif_make_list_cell(env, out_term2, list); > > enif_binary_to_term(env, bin.data, bin.size, &out_term3, 0); > list = enif_make_list_cell(env, out_term3, list); > > return list; > } > > The multiple calls to enif_binary_to_term somehow seem to corrupt memory > in the calling environment, so I'm probably using it incorrectly. Could > some kind soul point me to the error? I'm running this code on OTP20 / > erts-9.2. > > Thanks in advance, > Vincent > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamhedd@REDACTED Tue May 15 17:29:02 2018 From: jamhedd@REDACTED (Roman Galeev) Date: Tue, 15 May 2018 17:29:02 +0200 Subject: [erlang-questions] how to start and stop erlang daemon In-Reply-To: References: Message-ID: IMO an application should validate configuration itself and loudly refuse to start if the required configuration is inconsistent or absent. Daemons start/stop/control procedures afaik are defined by each and every Linux distro out there already (iirc mostly by guessing/checking pids), how beam.smp is different? On Tue, May 15, 2018 at 2:39 PM, Max Lapshin wrote: > Hi. > > > I've written a bit unstructured thoughts about seamless start and stop of > erlang daemon: > > https://medium.com/@max.lapshin/how-to-start-and-stop- > erlang-daemon-3fd988777ab3 > > In short: > > 1) validate config before launching > 2) control that daemon has started > 3) find a way to notify why it hasn't started > 4) use several ways to stop it and to ensure that it has stopped > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- With best regards, Roman Galeev, +420 702 817 968 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew@REDACTED Tue May 15 17:53:32 2018 From: andrew@REDACTED (Andrew Thompson) Date: Tue, 15 May 2018 11:53:32 -0400 Subject: [erlang-questions] Global lock is unfair? In-Reply-To: References: <843fe2aa-4533-bf8b-fd15-c258ec9b8ae3@gmail.com> Message-ID: <20180515155332.GD29047@thecloud.hijacked.us> On Mon, May 14, 2018 at 04:53:08PM -0500, Ryan Stewart wrote: > Funneling all the calls through a single process would bring the system to > a grinding halt. Sure, I could write a system of processes, each of which > acts as a mutex to one, particular resource, but why would I want to? All I > need is locking, just like global locks, but fair. It seems like something > that should already exist in a proven library. > For simple cases, wrapping a process around the individual resource and using it as a lock is actually fine and should scale well. If you want something fancier, there's Ulf's library: https://github.com/uwiger/locks Andrew From per@REDACTED Tue May 15 18:12:51 2018 From: per@REDACTED (Per Hedeland) Date: Tue, 15 May 2018 18:12:51 +0200 Subject: [erlang-questions] Handshake failure with DHE-DSS cipher suites In-Reply-To: References: <3e7e998e-a807-bb1a-dc0e-3ebbbe753fcd@hedeland.org> Message-ID: On 2018-05-15 15:08, Ingela Andin wrote: > Hi! > > I think that the oid is missing due to an oversight in adding it when crypto was updated to support it. For some reason (which I do not know) early versions of crypto did only > support sha1 (sha) with dsa. Afer further digging, I think it's just a matter of "evolution" - the 'id-dsa-with-sha1' oid is defined in asn1/PKIX1Algorithms88.asn1, but 'id-dsa-with-sha256' is a more recent thing, and neither defined there nor in any of the other modules i the asn1 directory, and I'm not sure it "belongs" in any of them. The definitions that can be found on the 'net agree on the numerical oid elements, but there are actually at least two variants on the (irrelevant here) naming of them - the one under "DSA with SHA-2 family" on https://csrc.nist.gov/Projects/Computer-Security-Objects-Register/Algorithm-Registration seems pretty authoritative to me:-), but it's not given as part of any ASN.1 module AFAIU. However https://www.itu.int/ITU-T/formal-language/itu-t/x/x509/2016/AlgorithmObjectIdentifiers.html seems to do that, with a module that also includes most but not all of the definitions that are handled by public_key:pkix_sign_types/1... > The correct oid name should be in the generated include file included in public_key.hrl Well it isn't, since it isn't defined in any of the modules in the asn1 directory. But anyway, just dropping the definition from above into the one of them that seems to have most of the definitions handled by pkix_sign_types/1 makes for a slighly nicer diff (below). > Without looking into this further, I suspect that the passing of the suites, could be due to that the actual sha function used is a result of the "hash_sign negotiation" that depends on the > hash_sign hello extension. I'm not worried by that:-), but I'd like to make the DHE-DSS suites work. I'll go with the below diff for now, but maybe OTP will fix it "properly" in some future version? --Per diff --git a/lib/public_key/asn1/PKCS-1.asn1 b/lib/public_key/asn1/PKCS-1.asn1 index 117eacd..1df6719 100644 --- a/lib/public_key/asn1/PKCS-1.asn1 +++ b/lib/public_key/asn1/PKCS-1.asn1 @@ -87,6 +87,12 @@ id-sha512 OBJECT IDENTIFIER ::= { joint-iso-itu-t(2) nistalgorithm(4) hashalgs(2) 3 } +-- This probably doesn't belong here, but... +id-dsa-with-sha256 OBJECT IDENTIFIER ::= { joint-iso-itu-t(2) + country(16) us(840) organization(1) gov(101) csor(3) + nistalgorithm(4) sigalgs(3) 2 } + + RSAPublicKey ::= SEQUENCE { modulus INTEGER, -- n publicExponent INTEGER -- e diff --git a/lib/public_key/src/public_key.erl b/lib/public_key/src/public_key.erl index 0341266..75305cb 100644 --- a/lib/public_key/src/public_key.erl +++ b/lib/public_key/src/public_key.erl @@ -498,6 +498,8 @@ pkix_sign_types(?'id-dsa-with-sha1') -> {sha, dsa}; pkix_sign_types(?'id-dsaWithSHA1') -> {sha, dsa}; +pkix_sign_types(?'id-dsa-with-sha256') -> + {sha256, dsa}; pkix_sign_types(?'ecdsa-with-SHA1') -> {sha, ecdsa}; pkix_sign_types(?'ecdsa-with-SHA256') -> From zzantozz@REDACTED Tue May 15 19:52:44 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Tue, 15 May 2018 12:52:44 -0500 Subject: [erlang-questions] Global lock is unfair? In-Reply-To: <20180515155332.GD29047@thecloud.hijacked.us> References: <843fe2aa-4533-bf8b-fd15-c258ec9b8ae3@gmail.com> <20180515155332.GD29047@thecloud.hijacked.us> Message-ID: Thanks for the pointer, Andrew. The process per resource would work, but how would you ensure that only one process is started per resource? The only way I know would be to write a central "manager" process that keeps track of the resource processes and directs traffic according to the resource "key", ensuring a new process is started when a new resource comes in demand. On top of that, the manager has to be aware of other nodes in the cluster and resource processes that might already exist elsewhere That's still a lot more work than seems necessary. On Tue, May 15, 2018 at 10:53 AM Andrew Thompson < andrew@REDACTED> wrote: > On Mon, May 14, 2018 at 04:53:08PM -0500, Ryan Stewart wrote: > > Funneling all the calls through a single process would bring the system > to > > a grinding halt. Sure, I could write a system of processes, each of which > > acts as a mutex to one, particular resource, but why would I want to? > All I > > need is locking, just like global locks, but fair. It seems like > something > > that should already exist in a proven library. > > > > For simple cases, wrapping a process around the individual resource and > using it as a lock is actually fine and should scale well. If you want > something fancier, there's Ulf's library: > https://github.com/uwiger/locks > > Andrew > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zzantozz@REDACTED Tue May 15 20:13:20 2018 From: zzantozz@REDACTED (Ryan Stewart) Date: Tue, 15 May 2018 13:13:20 -0500 Subject: [erlang-questions] Global lock is unfair? In-Reply-To: References: <843fe2aa-4533-bf8b-fd15-c258ec9b8ae3@gmail.com> <20180515155332.GD29047@thecloud.hijacked.us> Message-ID: Dan, thanks for the mnesia hint. It looks like a murky pool to dive into, but I'll read into it more. On Tue, May 15, 2018 at 12:52 PM Ryan Stewart wrote: > Thanks for the pointer, Andrew. The process per resource would work, but > how would you ensure that only one process is started per resource? The > only way I know would be to write a central "manager" process that keeps > track of the resource processes and directs traffic according to the > resource "key", ensuring a new process is started when a new resource comes > in demand. On top of that, the manager has to be aware of other nodes in > the cluster and resource processes that might already exist elsewhere > That's still a lot more work than seems necessary. > > On Tue, May 15, 2018 at 10:53 AM Andrew Thompson < > andrew@REDACTED> wrote: > >> On Mon, May 14, 2018 at 04:53:08PM -0500, Ryan Stewart wrote: >> > Funneling all the calls through a single process would bring the system >> to >> > a grinding halt. Sure, I could write a system of processes, each of >> which >> > acts as a mutex to one, particular resource, but why would I want to? >> All I >> > need is locking, just like global locks, but fair. It seems like >> something >> > that should already exist in a proven library. >> > >> >> For simple cases, wrapping a process around the individual resource and >> using it as a lock is actually fine and should scale well. If you want >> something fancier, there's Ulf's library: >> https://github.com/uwiger/locks >> >> Andrew >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue May 15 20:40:10 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 15 May 2018 20:40:10 +0200 Subject: [erlang-questions] Global lock is unfair? In-Reply-To: References: <843fe2aa-4533-bf8b-fd15-c258ec9b8ae3@gmail.com> <20180515155332.GD29047@thecloud.hijacked.us> Message-ID: On Tue, May 15, 2018 at 7:53 PM Ryan Stewart wrote: > Thanks for the pointer, Andrew. The process per resource would work, but > how would you ensure that only one process is started per resource? The > only way I know would be to write a central "manager" process that keeps > track of the resource processes and directs traffic according to the > resource "key", ensuring a new process is started when a new resource comes > in demand. On top of that, the manager has to be aware of other nodes in > the cluster and resource processes that might already exist elsewhere > That's still a lot more work than seems necessary. > > What are your failure semantics? One "solution" is to shard the key to a node, and register the process locally on that node. Since only one can be registered at a time, this work somewhat and provides a lock on the resource. I.e., just steal riak_core and use it :P Another solution, which you shouldn't dismiss, is to put all locks on one node only. This gives you a single-point-of-failure, but unless you need several 9's of uptime, the solution is easy to implement, easy to reason about and the 9's it gives you are very nice to have. Beware the advanced algorithm which has a bug because it can easily make your reliability measure in 9's _worse_ than having a deliberate S.P.O.F you understand. However, * Failure semantics matter. If one node goes down, all the locks on that node will be lost, etc. * Ulf Wiger's locks library should work in a distributed setting and solve this problem, but with a different set of failure semantics. Usually, a distributed lock that is truly resistant to node failure and so on is a hard problem. Theoretically, because just coming up with an algorithm is hard (Raft, PAXOS (multi-PAXOS, Fast-PAXOS, etc). Practically because implementing said algorithms in an error-free way is equally hard. If you haven't, the SRE Handbook by Google has a chapter[0] on this. It is a decent survey which gives you an overview of the problem space without delving too much into Erlang. Once you know what you are looking for and what tooling you have, it is usually easier to pick a solution. [0] https://landing.google.com/sre/book/chapters/managing-critical-state.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From torben.lehoff@REDACTED Tue May 15 22:59:50 2018 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Tue, 15 May 2018 22:59:50 +0200 Subject: [erlang-questions] gen_statem badrecord,trans_opts In-Reply-To: References: Message-ID: Hi, Would it be possible to add some text to the gen_statem docs that this message usually means you have messed up one of the actions? It wasn't obvious to me before I googled it and then it was a matter of chasing down all my actions. Somehow I find the error messages of gen_statem harder to work with than gen_fsm, which is a pity since gen_statem is so much easier to work with in terms of design flexibility and easy to raed code. Cheers, Torben On Tue, May 15, 2018 at 8:23 AM getonga wrote: > Thanks, I recheck my source code, and I found the error reason. Thanks for > your time. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- https://www.linkedin.com/in/torbenhoffmann/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Wed May 16 08:52:15 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 16 May 2018 08:52:15 +0200 Subject: [erlang-questions] gen_statem badrecord,trans_opts In-Reply-To: References: Message-ID: <20180516065215.GA44750@erix.ericsson.se> On Tue, May 15, 2018 at 10:59:50PM +0200, Torben Hoffmann wrote: > Hi, > Would it be possible to add some text to the gen_statem docs that this > message usually means you have messed up one of the actions? > It wasn't obvious to me before I googled it and then it was a matter of > chasing down all my actions. > Somehow I find the error messages of gen_statem harder to work with than > gen_fsm, which is a pity since gen_statem is so much easier to work with in > terms of design flexibility and easy to raed code. > > Cheers, > Torben The message in this case: Context: child_terminated Reason: {{badrecord,trans_opts}, [{gen_statem,loop_event_actions_list,10, [{file,"gen_statem.erl"},{line,1210}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,247}]}]} I classify this as a bug, but not severe enough to merit a patch on the 20.3 release. What the code was trying to do but crashed instead was to terminate with error:{bad_action_from_state_function,Action} in a regular Error Report. The intention is that gen_statem should pinpoint bad actions fairly precisely. That crash is fixed in 20.0-rc1 and later so it will terminate as intended. I agree that the OP:s crash is not a good error message for end users, but it was a good one for me since it was precise for me to fix the crash. Do you have any suggestions about improving correct error messages from gen_statem, or can you exemplify some that you find hard to understand? In comparison with gen_fsm, if you like. Best Regards / Raimo > > > On Tue, May 15, 2018 at 8:23 AM getonga wrote: > > > Thanks, I recheck my source code, and I found the error reason. Thanks for > > your time. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > https://www.linkedin.com/in/torbenhoffmann/ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From ingela.andin@REDACTED Wed May 16 10:21:15 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 16 May 2018 10:21:15 +0200 Subject: [erlang-questions] Handshake failure with DHE-DSS cipher suites In-Reply-To: References: <3e7e998e-a807-bb1a-dc0e-3ebbbe753fcd@hedeland.org> Message-ID: I think it belongs in diff --git a/lib/public_key/asn1/PKIX1Algorithms88.asn1 b/lib/public_key/asn1/PKIX1Algorithms88.asn1 index 6cc6745..13ac6fa 100644 --- a/lib/public_key/asn1/PKIX1Algorithms88.asn1 +++ b/lib/public_key/asn1/PKIX1Algorithms88.asn1 @@ -40,6 +40,12 @@ } -- encoding for DSA signature generated with SHA-1 hash +id-dsa-with-sha256 OBJECT IDENTIFIER ::= { joint-iso-itu-t(2) ++ country(16) us(840) organization(1) gov(101) csor(3) ++ nistalgorithm(4) sigalgs(3) 2 } ++ + + Dss-Sig-Value ::= SEQUENCE { r INTEGER, s INTEGER } You are welcome to submit a PR. Regards Ingela Erlang/OTP - Team Ericsson AB 2018-05-15 18:12 GMT+02:00 Per Hedeland : > On 2018-05-15 15:08, Ingela Andin wrote: > > Hi! > > > > I think that the oid is missing due to an oversight in adding it when > crypto was updated to support it. For some reason (which I do not know) > early versions of crypto did only > > support sha1 (sha) with dsa. > > Afer further digging, I think it's just a matter of "evolution" - the > 'id-dsa-with-sha1' oid is defined in asn1/PKIX1Algorithms88.asn1, but > 'id-dsa-with-sha256' is a more recent thing, and neither defined there > nor in any of the other modules i the asn1 directory, and I'm not sure > it "belongs" in any of them. > > The definitions that can be found on the 'net agree on the numerical oid > elements, but there are actually at least two variants on the > (irrelevant here) naming of them - the one under "DSA with SHA-2 family" > on > https://csrc.nist.gov/Projects/Computer-Security- > Objects-Register/Algorithm-Registration > seems pretty authoritative to me:-), but it's not given as part of any > ASN.1 module AFAIU. However > https://www.itu.int/ITU-T/formal-language/itu-t/x/x509/2016/ > AlgorithmObjectIdentifiers.html > seems to do that, with a module that also includes most but not all of > the definitions that are handled by public_key:pkix_sign_types/1... > > > The correct oid name should be in the generated include file included in > public_key.hrl > > Well it isn't, since it isn't defined in any of the modules in the asn1 > directory. But anyway, just dropping the definition from above into the > one of them that seems to have most of the definitions handled by > pkix_sign_types/1 makes for a slighly nicer diff (below). > > > Without looking into this further, I suspect that the passing of the > suites, could be due to that the actual sha function used is a result of > the "hash_sign negotiation" that depends on the > > hash_sign hello extension. > > I'm not worried by that:-), but I'd like to make the DHE-DSS suites > work. I'll go with the below diff for now, but maybe OTP will fix it > "properly" in some future version? > > --Per > > diff --git a/lib/public_key/asn1/PKCS-1.asn1 b/lib/public_key/asn1/PKCS-1. > asn1 > index 117eacd..1df6719 100644 > --- a/lib/public_key/asn1/PKCS-1.asn1 > +++ b/lib/public_key/asn1/PKCS-1.asn1 > @@ -87,6 +87,12 @@ id-sha512 OBJECT IDENTIFIER ::= { joint-iso-itu-t(2) > nistalgorithm(4) hashalgs(2) 3 } > > > +-- This probably doesn't belong here, but... > +id-dsa-with-sha256 OBJECT IDENTIFIER ::= { joint-iso-itu-t(2) > + country(16) us(840) organization(1) gov(101) csor(3) > + nistalgorithm(4) sigalgs(3) 2 } > + > + > RSAPublicKey ::= SEQUENCE { > modulus INTEGER, -- n > publicExponent INTEGER -- e > diff --git a/lib/public_key/src/public_key.erl > b/lib/public_key/src/public_key.erl > index 0341266..75305cb 100644 > --- a/lib/public_key/src/public_key.erl > +++ b/lib/public_key/src/public_key.erl > @@ -498,6 +498,8 @@ pkix_sign_types(?'id-dsa-with-sha1') -> > {sha, dsa}; > pkix_sign_types(?'id-dsaWithSHA1') -> > {sha, dsa}; > +pkix_sign_types(?'id-dsa-with-sha256') -> > + {sha256, dsa}; > pkix_sign_types(?'ecdsa-with-SHA1') -> > {sha, ecdsa}; > pkix_sign_types(?'ecdsa-with-SHA256') -> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From torben.lehoff@REDACTED Wed May 16 10:39:36 2018 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Wed, 16 May 2018 10:39:36 +0200 Subject: [erlang-questions] gen_statem badrecord,trans_opts In-Reply-To: <20180516065215.GA44750@erix.ericsson.se> References: <20180516065215.GA44750@erix.ericsson.se> Message-ID: Hi Raimo, I'm looking forward to that particular error being more understandable in a future release. It is too small to patch for. The other day I was trying to capture a somewhat elaborate state machine in a gen_statem and I hadn't read the manual well enough, so I thought that I could do some nifty checking in the enter clause and decide if another state was more suited to do the job. But with state_enter you cannot change state in the enter clause. It is clearly written din the manual, but... you know... blush. However, the error message doesn't point you to this fundamental rule: ** Last event = {internal,init_state} ** When server state = {starting,happy} ** Reason for termination = error:{bad_return_from_state_function, {next_state,creating,happy}} ** Callback mode = [state_functions,state_enter] It would be nice if it said something about not changing state when entering. Same kind of issue when adding next_event in a state enter: ** Reason for termination = error:{bad_action_from_state_function, {next_event,internal,jump}} Maybe the error should be {dont_add_next_event_in_enter_read_the_manual_dude, ...} or something to that effect! And when you misspell next_state you get this: ** Reason for termination = error:{bad_return_from_state_function, {next_stete,running,happy, {next_event,internal,jump}}} It would be nice if it said what it could not like. I have not looked in the gen_statem code, but I am pretty sure that the checks would allow more details in the error messages. The information must be at hand when it errors out. Cheers, Torben On Wed, May 16, 2018 at 8:52 AM Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > On Tue, May 15, 2018 at 10:59:50PM +0200, Torben Hoffmann wrote: > > Hi, > > Would it be possible to add some text to the gen_statem docs that this > > message usually means you have messed up one of the actions? > > It wasn't obvious to me before I googled it and then it was a matter of > > chasing down all my actions. > > Somehow I find the error messages of gen_statem harder to work with than > > gen_fsm, which is a pity since gen_statem is so much easier to work with > in > > terms of design flexibility and easy to raed code. > > > > Cheers, > > Torben > > The message in this case: > Context: child_terminated Reason: > {{badrecord,trans_opts}, > [{gen_statem,loop_event_actions_list,10, > [{file,"gen_statem.erl"},{line,1210}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,247}]}]} > > I classify this as a bug, but not severe enough to merit a patch on the > 20.3 > release. What the code was trying to do but crashed instead was to > terminate with error:{bad_action_from_state_function,Action} in a regular > Error Report. > > The intention is that gen_statem should pinpoint bad actions fairly > precisely. That crash is fixed in 20.0-rc1 and later so it will terminate > as intended. > > I agree that the OP:s crash is not a good error message for end users, but > it was a good one for me since it was precise for me to fix the crash. > > Do you have any suggestions about improving correct error messages from > gen_statem, or can you exemplify some that you find hard to understand? > In comparison with gen_fsm, if you like. > > Best Regards > / Raimo > > > > > > > > On Tue, May 15, 2018 at 8:23 AM getonga wrote: > > > > > Thanks, I recheck my source code, and I found the error reason. Thanks > for > > > your time. > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -- > > https://www.linkedin.com/in/torbenhoffmann/ > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- https://www.linkedin.com/in/torbenhoffmann/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Wed May 16 12:05:09 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 16 May 2018 12:05:09 +0200 Subject: [erlang-questions] New SSL option to set TLS record version? In-Reply-To: References: Message-ID: Hi! It would be thinkable to have such an option for introp reasons. PR are welcome. Regards Ingela Erlang/OTP Team - Ericsson AB 2018-05-09 17:30 GMT+02:00 Ryan Stewart : > I've run across a faulty SSL server implementation that appears to send a > "handshake failure" alert if the ClientHello protocol version isn't equal > to the TLS record version. In Erlang, different major versions choose the > TLS record version differently. None of them are wrong according the TLS > spec, but some of them break when I'm trying to connect to these bad server > implementations. > > What do you think of adding a new ssl_option like "client_hello_tls_record_version" > to let us explicitly set the version to be used? Ideally, it would support > values like 'tlsv1', 'tlsv1_2', 'lowest', 'highest', and > 'same_as_client_hello', for example. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Wed May 16 12:13:28 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 16 May 2018 12:13:28 +0200 Subject: [erlang-questions] gen_statem badrecord,trans_opts In-Reply-To: References: <20180516065215.GA44750@erix.ericsson.se> Message-ID: <20180516101328.GA65066@erix.ericsson.se> On Wed, May 16, 2018 at 10:39:36AM +0200, Torben Hoffmann wrote: > Hi Raimo, > > I'm looking forward to that particular error being more understandable in a > future release. It is too small to patch for. > > The other day I was trying to capture a somewhat elaborate state machine in > a gen_statem and I hadn't read the manual well enough, so I thought that I > could do some nifty checking in the enter clause and decide if another > state was more suited to do the job. But with state_enter you cannot change > state in the enter clause. It is clearly written din the manual, but... you > know... blush. > > However, the error message doesn't point you to this fundamental rule: > > ** Last event = {internal,init_state} > ** When server state = {starting,happy} > ** Reason for termination = error:{bad_return_from_state_function, > {next_state,creating,happy}} > ** Callback mode = [state_functions,state_enter] > > It would be nice if it said something about not changing state when > entering. That one was changed to error:{bad_state_enter_return_from_state_function, {next_state,creating,happy}} in commit ea1963553ffb06eb1bda636f328718f91136ed9c on 'master' I hope that is not too subtle - I wanted to stick to the format. At least it indicates state_enter and state function. > > Same kind of issue when adding next_event in a state enter: > > ** Reason for termination = error:{bad_action_from_state_function, > {next_event,internal,jump}} > > Maybe the error should be > {dont_add_next_event_in_enter_read_the_manual_dude, ...} or something to > that effect! > That one was changed to error:{bad_state_enter_action_from_state_function, {next_event,internal,jump}} in the same commit. > And when you misspell next_state you get this: > > ** Reason for termination = error:{bad_return_from_state_function, > {next_stete,running,happy, > {next_event,internal,jump}}} > > It would be nice if it said what it could not like. In this case you return a tuple with a tag that is not recognized, and it would take some artificial intelligence to figure out that it most likely should be next_state that the user attempted. The tuple does not match any of the allowed patterns, simply, so it is a bad return value. Difficult to do anything smarter. Best regards / Raimo > > I have not looked in the gen_statem code, but I am pretty sure that the > checks would allow more details in the error messages. The information must > be at hand when it errors out. > > Cheers, > Torben > > > On Wed, May 16, 2018 at 8:52 AM Raimo Niskanen < > raimo+erlang-questions@REDACTED> wrote: > > > On Tue, May 15, 2018 at 10:59:50PM +0200, Torben Hoffmann wrote: > > > Hi, > > > Would it be possible to add some text to the gen_statem docs that this > > > message usually means you have messed up one of the actions? > > > It wasn't obvious to me before I googled it and then it was a matter of > > > chasing down all my actions. > > > Somehow I find the error messages of gen_statem harder to work with than > > > gen_fsm, which is a pity since gen_statem is so much easier to work with > > in > > > terms of design flexibility and easy to raed code. > > > > > > Cheers, > > > Torben > > > > The message in this case: > > Context: child_terminated Reason: > > {{badrecord,trans_opts}, > > [{gen_statem,loop_event_actions_list,10, > > [{file,"gen_statem.erl"},{line,1210}]}, > > {proc_lib,init_p_do_apply,3, > > [{file,"proc_lib.erl"},{line,247}]}]} > > > > I classify this as a bug, but not severe enough to merit a patch on the > > 20.3 > > release. What the code was trying to do but crashed instead was to > > terminate with error:{bad_action_from_state_function,Action} in a regular > > Error Report. > > > > The intention is that gen_statem should pinpoint bad actions fairly > > precisely. That crash is fixed in 20.0-rc1 and later so it will terminate > > as intended. > > > > I agree that the OP:s crash is not a good error message for end users, but > > it was a good one for me since it was precise for me to fix the crash. > > > > Do you have any suggestions about improving correct error messages from > > gen_statem, or can you exemplify some that you find hard to understand? > > In comparison with gen_fsm, if you like. > > > > Best Regards > > / Raimo > > > > > > > > > > > > > On Tue, May 15, 2018 at 8:23 AM getonga wrote: > > > > > > > Thanks, I recheck my source code, and I found the error reason. Thanks > > for > > > > your time. > > > -- > > > https://www.linkedin.com/in/torbenhoffmann/ > > > > -- > > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > -- > https://www.linkedin.com/in/torbenhoffmann/ -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From torben.lehoff@REDACTED Wed May 16 13:30:16 2018 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Wed, 16 May 2018 13:30:16 +0200 Subject: [erlang-questions] gen_statem badrecord,trans_opts In-Reply-To: <20180516101328.GA65066@erix.ericsson.se> References: <20180516065215.GA44750@erix.ericsson.se> <20180516101328.GA65066@erix.ericsson.se> Message-ID: Those improvements are really good. I can live without AI in the OTP libs ;-) Cheers, Torben On Wed, May 16, 2018 at 12:13 PM Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > On Wed, May 16, 2018 at 10:39:36AM +0200, Torben Hoffmann wrote: > > Hi Raimo, > > > > I'm looking forward to that particular error being more understandable > in a > > future release. It is too small to patch for. > > > > The other day I was trying to capture a somewhat elaborate state machine > in > > a gen_statem and I hadn't read the manual well enough, so I thought that > I > > could do some nifty checking in the enter clause and decide if another > > state was more suited to do the job. But with state_enter you cannot > change > > state in the enter clause. It is clearly written din the manual, but... > you > > know... blush. > > > > However, the error message doesn't point you to this fundamental rule: > > > > ** Last event = {internal,init_state} > > ** When server state = {starting,happy} > > ** Reason for termination = error:{bad_return_from_state_function, > > {next_state,creating,happy}} > > ** Callback mode = [state_functions,state_enter] > > > > It would be nice if it said something about not changing state when > > entering. > > That one was changed to > error:{bad_state_enter_return_from_state_function, > {next_state,creating,happy}} > in commit ea1963553ffb06eb1bda636f328718f91136ed9c on 'master' > > I hope that is not too subtle - I wanted to stick to the format. > At least it indicates state_enter and state function. > > > > > > Same kind of issue when adding next_event in a state enter: > > > > ** Reason for termination = error:{bad_action_from_state_function, > > {next_event,internal,jump}} > > > > Maybe the error should be > > {dont_add_next_event_in_enter_read_the_manual_dude, ...} or something to > > that effect! > > > > That one was changed to > error:{bad_state_enter_action_from_state_function, > {next_event,internal,jump}} > in the same commit. > > > > And when you misspell next_state you get this: > > > > ** Reason for termination = error:{bad_return_from_state_function, > > {next_stete,running,happy, > > {next_event,internal,jump}}} > > > > It would be nice if it said what it could not like. > > In this case you return a tuple with a tag that is not recognized, and it > would take some artificial intelligence to figure out that it most likely > should be next_state that the user attempted. > > The tuple does not match any of the allowed patterns, simply, so it is > a bad return value. Difficult to do anything smarter. > > > Best regards > / Raimo > > > > > > I have not looked in the gen_statem code, but I am pretty sure that the > > checks would allow more details in the error messages. The information > must > > be at hand when it errors out. > > > > Cheers, > > Torben > > > > > > On Wed, May 16, 2018 at 8:52 AM Raimo Niskanen < > > raimo+erlang-questions@REDACTED> wrote: > > > > > On Tue, May 15, 2018 at 10:59:50PM +0200, Torben Hoffmann wrote: > > > > Hi, > > > > Would it be possible to add some text to the gen_statem docs that > this > > > > message usually means you have messed up one of the actions? > > > > It wasn't obvious to me before I googled it and then it was a matter > of > > > > chasing down all my actions. > > > > Somehow I find the error messages of gen_statem harder to work with > than > > > > gen_fsm, which is a pity since gen_statem is so much easier to work > with > > > in > > > > terms of design flexibility and easy to raed code. > > > > > > > > Cheers, > > > > Torben > > > > > > The message in this case: > > > Context: child_terminated Reason: > > > {{badrecord,trans_opts}, > > > [{gen_statem,loop_event_actions_list,10, > > > [{file,"gen_statem.erl"},{line,1210}]}, > > > {proc_lib,init_p_do_apply,3, > > > [{file,"proc_lib.erl"},{line,247}]}]} > > > > > > I classify this as a bug, but not severe enough to merit a patch on the > > > 20.3 > > > release. What the code was trying to do but crashed instead was to > > > terminate with error:{bad_action_from_state_function,Action} in a > regular > > > Error Report. > > > > > > The intention is that gen_statem should pinpoint bad actions fairly > > > precisely. That crash is fixed in 20.0-rc1 and later so it will > terminate > > > as intended. > > > > > > I agree that the OP:s crash is not a good error message for end users, > but > > > it was a good one for me since it was precise for me to fix the crash. > > > > > > Do you have any suggestions about improving correct error messages from > > > gen_statem, or can you exemplify some that you find hard to understand? > > > In comparison with gen_fsm, if you like. > > > > > > Best Regards > > > / Raimo > > > > > > > > > > > > > > > > > > On Tue, May 15, 2018 at 8:23 AM getonga wrote: > > > > > > > > > Thanks, I recheck my source code, and I found the error reason. > Thanks > > > for > > > > > your time. > > > > -- > > > > https://www.linkedin.com/in/torbenhoffmann/ > > > > > > -- > > > > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > > -- > > https://www.linkedin.com/in/torbenhoffmann/ > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- https://www.linkedin.com/in/torbenhoffmann/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From prakash.parmar@REDACTED Wed May 16 14:37:21 2018 From: prakash.parmar@REDACTED (Prakash Parmar) Date: Wed, 16 May 2018 12:37:21 +0000 Subject: [erlang-questions] Mnesia Replication with local content Message-ID: Hello All, According to my project requirements, I have Two mnesia nodes namely 'node1@REDACTED' and 'node2@REDACTED'. I want to create a Mnesia table with local content and same table name on both nodes. Here are the Steps I'm following : In One Terminal - $ erl -sname node1 -setcookie abc On another Terminal - $ erl -sname node2 -setcookie abc ===================================================================================== (node1@REDACTED)1> net_adm:ping('node2@REDACTED'). pong (node1@REDACTED)2> nodes(). [node2@REDACTED] (node1@REDACTED)3> mnesia:create_schema(['node1@REDACTED', 'node2@REDACTED']). ok (node1@REDACTED)4> rpc:multicall(['node1@REDACTED', 'node2@REDACTED'], mnesia, start,[]). {[ok,ok],[]} (node1@REDACTED)5> mnesia:create_table( log_count,[{attributes, [counter, value]},{local_content, true},{disc_copies, ['node1@REDACTED','node2@REDACTED']}]). {atomic,ok} (node1@REDACTED)6> mnesia:info(). ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- log_count : with 0 records occupying 298 words of mem schema : with 2 records occupying 530 words of mem ===> System info in version "4.14.3", debug level = none <=== opt_disc. Directory "/home/user/svn/mvpn-1.1/Mnesia.node1@REDACTED" is used. use fallback at restart = false running db nodes = [node2@REDACTED,node1@REDACTED] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [log_count,schema] disc_only_copies = [] [{node1@REDACTED,disc_copies}] = [log_count] [{node1@REDACTED,disc_copies},{node2@REDACTED,disc_copies}] = [schema] 4 transactions committed, 2 aborted, 0 restarted, 5 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] ok ===================================================================================== Can anyone help me out why Table is not getting created on a 2ndnode? When I removed option {local_content, true} that table getting created on a 2nd node. Is there another way to achive this? /Prakash Parmar -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Wed May 16 14:43:22 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 16 May 2018 14:43:22 +0200 Subject: [erlang-questions] Erlang/OTP 21.0-rc1 (Release Candidate) In-Reply-To: References: <1525261079.4069.73.camel@ericsson.com> <00f10191-7c69-a2ba-3749-97ac4ce5a984@ninenines.eu> <75296de9-612f-ab91-1c09-0379f77e6a87@ninenines.eu> Message-ID: Hi! Finally a solution: https://github.com/erlang/otp/pull/1820 Regards Ingela Erlang/OTP team - Ericsson AB 2018-05-07 8:55 GMT+02:00 Ingela Andin : > Hi again! > > I think I need to take back that last change, it is a bit more > complicated. I will work on a real fix during the week. > > Regards Ingela Erlang/OTP Team > > > 2018-05-04 19:18 GMT+02:00 Ingela Andin : > >> I hope this hole fix please try it out: >> >> diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl >> index 0956d35..ed8663b 100644 >> --- a/lib/ssl/src/ssl_cipher.erl >> +++ b/lib/ssl/src/ssl_cipher.erl >> @@ -2837,11 +2837,13 @@ key_uses(OtpCert) -> >> Extensions = ssl_certificate:extensions_list(TBSExtensions), >> case ssl_certificate:select_extension(?'id-ce-keyUsage', >> Extensions) of >> undefined -> >> - undefined; >> + []; >> #'Extension'{extnValue = KeyUses} -> >> KeyUses >> end. >> >> +filter_keyuse_suites(_, [], CipherSuits, _) -> >> + CipherSuits; >> filter_keyuse_suites(Use, KeyUse, CipherSuits, Suites) -> >> case ssl_certificate:is_valid_key_usage(KeyUse, Use) of >> true -> >> >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> >> >> 2018-05-04 17:41 GMT+02:00 Ingela Andin : >> >>> Hi! >>> >>> Thank you for the example >>> >>> I did find one bug, the patch is here: >>> >>> diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl >>> index 0956d35..dd3dc54 100644 >>> --- a/lib/ssl/src/ssl_cipher.erl >>> +++ b/lib/ssl/src/ssl_cipher.erl >>> @@ -2837,7 +2837,7 @@ key_uses(OtpCert) -> >>> Extensions = ssl_certificate:extensions_list(TBSExtensions), >>> case ssl_certificate:select_extension(?'id-ce-keyUsage', >>> Extensions) of >>> undefined -> >>> - undefined; >>> + []; >>> #'Extension'{extnValue = KeyUses} -> >>> KeyUses >>> end. >>> >>> >>> my other sslv2 issue seems not to be related. (Probably a OpenSSL issue) >>> Your example does however not work perfect >>> so I am continuing to look into this! >>> >>> Regards Ingela Erlang/OTP team - Ericsson AB >>> >>> >>> >>> >>> >>> 2018-05-04 12:25 GMT+02:00 Roger Lipscombe : >>> >>>> On 4 May 2018 at 08:32, Ingela Andin wrote: >>>> > This error is consistent with one of the errors I am seeing in the >>>> nightly >>>> > builds when running OpenSSL with only default parameters so I suspect >>>> > something is off in combination >>>> > version negotiation and cipher suite selection checks. I am looking >>>> in to >>>> > it! >>>> >>>> I'm seeing the same, if it helps to reproduce. I generated my >>>> certificates with: >>>> >>>> #!/bin/sh >>>> >>>> # Create the CA key and certificate. >>>> openssl genrsa -out ca.key 2048 >>>> openssl req -new -x509 -nodes -key ca.key -days 3653 -out ca.pem -subj >>>> "/CN=Test CA" >>>> >>>> # Create the server key and CSR. >>>> openssl genrsa -out server.key 2048 >>>> openssl req -new -key server.key -out server.csr -subj "/CN=localhost" >>>> >>>> # Sign the CSR with the CA key. >>>> serial=$(date +"%s") >>>> openssl x509 -req -days 3563 -CA ca.pem -CAserial $serial >>>> -CAcreateserial -CAkey ca.key -in server.csr -out server.pem >>>> rm $serial >>>> >>>> I tested with: >>>> >>>> % Server >>>> {ok, _} = application:ensure_all_started(ssl). >>>> Port = 11002. >>>> LOpts = [{certfile, "server.pem"}, {keyfile, "server.key"}]. >>>> {ok, LSock} = ssl:listen(Port, LOpts). >>>> {ok, CSock} = ssl:transport_accept(LSock). >>>> ok = ssl:ssl_accept(CSock). >>>> >>>> % Client >>>> {ok, _} = application:ensure_all_started(ssl). >>>> Port = 11002. >>>> COpts = [{verify, verify_peer}, {cacertfile, "ca.pem"}]. >>>> {ok, Sock} = ssl:connect("localhost", Port, COpts). >>>> >>>> The server reports: >>>> >>>> =INFO REPORT==== 4-May-2018::11:22:20.971130 === >>>> TLS server: In state hello at tls_handshake.erl:130 generated SERVER >>>> ALERT: Fatal - Handshake Failure - malformed_handshake_data >>>> >>>> ** exception error: no match of right hand side value >>>> {error,{tls_alert,"handshake failure"}} >>>> >>>> The client reports: >>>> >>>> =INFO REPORT==== 4-May-2018::11:22:20.981524 === >>>> TLS client: In state hello received SERVER ALERT: Fatal - Handshake >>>> Failure >>>> >>>> ** exception error: no match of right hand side value >>>> {error,{tls_alert,"handshake failure"}} >>>> >>>> The same code works fine with 20.3.1 >>>> >>>> Thanks, >>>> Roger. >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Wed May 16 14:51:22 2018 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Wed, 16 May 2018 12:51:22 +0000 Subject: [erlang-questions] Issue with using enif_binary_to_term In-Reply-To: References: Message-ID: <1526475082.8310.12.camel@ericsson.com> On tis, 2018-05-15 at 17:02 +0200, Vincent Siliakus wrote: > Hi Sverker, > > Thanks for the reply including the fix. Will there be an OTP 20 release that > will include this patch, or will it only be included in OTP 21? For internal > usage it doesn't really matter, but I'm hoping to release the library I'm > working on as open source in the near future. > There is no OTP-20.4 planned, but I've put the fix in the pipe to be included in the next patch release on OTP-20.3.* if (or dare I say when) that happens. /Sverker From max.lapshin@REDACTED Wed May 16 15:33:22 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 16 May 2018 16:33:22 +0300 Subject: [erlang-questions] how to start and stop erlang daemon In-Reply-To: References: Message-ID: This is what have I written: validate and loudly refuse to start before it is daemonized -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangud@REDACTED Wed May 16 15:38:06 2018 From: dangud@REDACTED (Dan Gudmundsson) Date: Wed, 16 May 2018 15:38:06 +0200 Subject: [erlang-questions] Mnesia Replication with local content In-Reply-To: References: Message-ID: If you do mnesia:info() on node2@REDACTED, it will list the table 'log_count' there as well. Each node have a "local" 'log_count' table that is not visible on the other nodes. On Wed, May 16, 2018 at 2:37 PM Prakash Parmar wrote: > Hello All, > > According to my project requirements, I have Two mnesia nodes namely > 'node1@REDACTED' and 'node2@REDACTED'. I want to create a Mnesia table with > local content and same table name on both nodes. > > Here are the Steps I'm following : > > In One Terminal - $ erl -sname node1 -setcookie abc > > On another Terminal - $ erl -sname node2 -setcookie abc > > > ===================================================================================== > > (node1@REDACTED)1> net_adm:ping('node2@REDACTED'). > pong > > (node1@REDACTED)2> nodes(). > [node2@REDACTED] > > (node1@REDACTED)3> mnesia:create_schema(['node1@REDACTED', 'node2@REDACTED > ']). > ok > > (node1@REDACTED)4> rpc:multicall(['node1@REDACTED', 'node2@REDACTED'], > mnesia, start,[]). > {[ok,ok],[]} > > (node1@REDACTED)5> mnesia:create_table( log_count,[{attributes, [counter, > value]},{local_content, true},{disc_copies, ['node1@REDACTED > ','node2@REDACTED']}]). > {atomic,ok} > > (node1@REDACTED)6> > mnesia:info(). > ---> Processes holding locks <--- > ---> Processes waiting for locks <--- > ---> Participant transactions <--- > ---> Coordinator transactions <--- > ---> Uncertain transactions <--- > ---> Active tables <--- > log_count : with 0 records occupying 298 words of mem > schema : with 2 records occupying 530 words of mem > ===> System info in version "4.14.3", debug level = none <=== > opt_disc. Directory "/home/user/svn/mvpn-1.1/Mnesia.node1@REDACTED" is > used. > use fallback at restart = false > running db nodes = [node2@REDACTED,node1@REDACTED] > stopped db nodes = [] > master node tables = [] > remote = [] > ram_copies = [] > disc_copies = [log_count,schema] > disc_only_copies = [] > [{node1@REDACTED,disc_copies}] = [log_count] > [{node1@REDACTED,disc_copies},{node2@REDACTED,disc_copies}] = [schema] > 4 transactions committed, 2 aborted, 0 restarted, 5 logged to disc > 0 held locks, 0 in queue; 0 local transactions, 0 remote > 0 transactions waits for other nodes: [] > ok > > ===================================================================================== > > Can anyone help me out why Table is not getting created on a 2ndnode? > > When I removed option {local_content, true} that table getting created on > a 2nd node. > > Is there another way to achive this? > > /Prakash Parmar > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamhedd@REDACTED Wed May 16 16:41:40 2018 From: jamhedd@REDACTED (Roman Galeev) Date: Wed, 16 May 2018 16:41:40 +0200 Subject: [erlang-questions] how to start and stop erlang daemon In-Reply-To: References: Message-ID: Well, my point is that the application should validate its own config, but not an external tool/script, as you wrote in your medium article: ./contrib/validate_config.erl || exit $? On Wed, May 16, 2018 at 3:33 PM, Max Lapshin wrote: > > This is what have I written: validate and loudly refuse to start before > it is daemonized > > -- With best regards, Roman Galeev, +420 702 817 968 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Wed May 16 18:44:06 2018 From: bchesneau@REDACTED (Benoit Chesneau) Date: Wed, 16 May 2018 18:44:06 +0200 Subject: [erlang-questions] webrtc lib Message-ID: Hi all, I was about to wrap janus in an erlang app to be able to communicate with browsers, but then I wondered if there isn't a "native" implementation these days ? Also I couldn't find a wrapper but maybe someone already wrote one? The main goal of using webrtc there is mostly beeing able to handle some UI in a P2P manner. Benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Wed May 16 23:00:40 2018 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Wed, 16 May 2018 23:00:40 +0200 Subject: [erlang-questions] webrtc lib In-Reply-To: References: Message-ID: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> > 16 maj 2018 kl. 18:44 skrev Benoit Chesneau : > > Hi all, > > I was about to wrap janus in an erlang app to be able to communicate with browsers, but then I wondered if there isn't a "native" implementation these days ? Also I couldn't find a wrapper but maybe someone already wrote one? I am not sure what you mean with ?wrapper? and ?wrapping? but if you mean to somehow run Janus inside Erlang or linking to it I have not done that and I am not sure I understand why one would do that. I have run Janus with an Erlang node controlling it but that Erlang node is just using Janus? JSONRPC interface over Websocket which is pretty straight forward to implement in Erlang (and any other language). I don?t have much code that I can share from this though sadly and I only implemented the few Janus calls that I needed (setting of calls in audiobridge and videoroom plugins) but it worked ok and I am happy to assist if you have questions. Janus has some hiccups I must say - there are some segfaults now and then depending on what plugins you use (see the issues on github for instance) but if you spread the calls over many Janus instances a crash will only affect a small set of users at least and not your entire user base. I could also not find the docs for the actual calls towards the plugins so you end up opening the plugin code to understand what parameters to include in the rpc calls. Stefan > > The main goal of using webrtc there is mostly beeing able to handle some UI in a P2P manner. > > Benoit > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bchesneau@REDACTED Thu May 17 10:41:11 2018 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 17 May 2018 10:41:11 +0200 Subject: [erlang-questions] webrtc lib In-Reply-To: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: Thanks for the infos. I agrre and was thinking to use the wa transport . But the main idea in wrapping it is to be able to ship it as a standalone software without relying on a container thing. Of course having a native way to it in erlang would be better :) Beno?t On Wednesday, May 16, 2018, Stefan Hellkvist wrote: > > > > 16 maj 2018 kl. 18:44 skrev Benoit Chesneau : > > > > Hi all, > > > > I was about to wrap janus in an erlang app to be able to communicate > with browsers, but then I wondered if there isn't a "native" implementation > these days ? Also I couldn't find a wrapper but maybe someone already wrote > one? > > I am not sure what you mean with ?wrapper? and ?wrapping? but if you mean > to somehow run Janus inside Erlang or linking to it I have not done that > and I am not sure I understand why one would do that. > > I have run Janus with an Erlang node controlling it but that Erlang node > is just using Janus? JSONRPC interface over Websocket which is pretty > straight forward to implement in Erlang (and any other language). I don?t > have much code that I can share from this though sadly and I only > implemented the few Janus calls that I needed (setting of calls in > audiobridge and videoroom plugins) but it worked ok and I am happy to > assist if you have questions. > > Janus has some hiccups I must say - there are some segfaults now and then > depending on what plugins you use (see the issues on github for instance) > but if you spread the calls over many Janus instances a crash will only > affect a small set of users at least and not your entire user base. I could > also not find the docs for the actual calls towards the plugins so you end > up opening the plugin code to understand what parameters to include in the > rpc calls. > > Stefan > > > > > > > The main goal of using webrtc there is mostly beeing able to handle some > UI in a P2P manner. > > > > Benoit > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > -- Sent from my Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryanm@REDACTED Thu May 17 11:08:25 2018 From: ryanm@REDACTED (Ryan Maclear) Date: Thu, 17 May 2018 11:08:25 +0200 Subject: [erlang-questions] proper char() type In-Reply-To: References: Message-ID: Appologies, I hit send before finisinh the email. Here is the complete mail: I'm busy learning to use proper and have the following issue. When calling the function >From the API docs, the char() type has a range of 0 to (1114111) 16#10ffff. When calling the function proper_gen:pick(proper_types:term()). manually many times (I've tried in around 3000 times) and on different VMs, the function always seems to return only one of the following values: {ok,0}, {ok,2}, {ok,3}, {ok,6} {ok,12} and {ok,13} I see the same behaviour for proper_gen:pick(proper_types:range(0,16#10ffff)). which I believe is equivalent. I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a ubuntu trusty docker image with erlang 19.3. In all three vms I see the same set of results being returned. Is there something else I need to do before calling this function? Thanks, Ryan On 17 May 2018 at 11:07, Ryan Maclear wrote: > Good Day, > > I'm busy learning to use proper and have the following issue. When calling > the function > > From the API docs, the char() type has a range of 0 to (1114111) 16#10ffff. > > When calling the function > > proper_gen:pick(proper_types:term()). > > manually many times (I've tried in around 3000 times) and on different > VMs, the function always seems to return only one of the following values: > > {ok,0}, > {ok,2}, > {ok,3}, > {ok,6} > {ok,12} and > {ok,13} > > I see the same behaviour for > > > I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a > ubuntu trusty docker image with erlang 19.3. > > In all three vms I see the same set of results being returned. > > Is there something else I need to do before calling this function? > > Thanks, > Ryan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu May 17 12:02:08 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 17 May 2018 12:02:08 +0200 Subject: [erlang-questions] proper char() type In-Reply-To: References: Message-ID: Most QuickCheck systems have an internal size parameter which is slowly increased over the course of testing. If you manually bump it by a lot, you get a better range: 16> proper_gen:pick(proper_types:resize(8000, proper_types:integer(0, 16#10ffff))). {ok,5050} However, I feel most of propers generators are botched and don't really check what they should. EQC Mini for instance: 5> eqc_gen:sample(eqc_gen:choose(0, 16#10ffff)). 482145 1062500 211296 269049 863625 4346 441921 719972 62442 703915 944194 ok which is far more the range you would expect. Also note that the pick/1 function is kind of dangerous since it leads people down a path where their generators have less randomness than they should have. The right solution often involves ?LET(X, generator(), <[X] E x>) and friends. On Thu, May 17, 2018 at 11:34 AM Ryan Maclear wrote: > Appologies, I hit send before finisinh the email. Here is the complete > mail: > > I'm busy learning to use proper and have the following issue. When calling > the function > > From the API docs, the char() type has a range of 0 to (1114111) 16#10ffff. > > When calling the function > > proper_gen:pick(proper_types:term()). > > manually many times (I've tried in around 3000 times) and on different > VMs, the function always seems to return only one of the following values: > > {ok,0}, > {ok,2}, > {ok,3}, > {ok,6} > {ok,12} and > {ok,13} > > I see the same behaviour for > > proper_gen:pick(proper_types:range(0,16#10ffff)). > > which I believe is equivalent. > > I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a > ubuntu trusty docker image with erlang 19.3. > > In all three vms I see the same set of results being returned. > > Is there something else I need to do before calling this function? > > Thanks, > Ryan > > > On 17 May 2018 at 11:07, Ryan Maclear wrote: > >> Good Day, >> >> I'm busy learning to use proper and have the following issue. When >> calling the function >> >> From the API docs, the char() type has a range of 0 to (1114111) >> 16#10ffff. >> >> When calling the function >> >> proper_gen:pick(proper_types:term()). >> >> manually many times (I've tried in around 3000 times) and on different >> VMs, the function always seems to return only one of the following values: >> >> {ok,0}, >> {ok,2}, >> {ok,3}, >> {ok,6} >> {ok,12} and >> {ok,13} >> >> I see the same behaviour for >> >> >> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >> ubuntu trusty docker image with erlang 19.3. >> >> In all three vms I see the same set of results being returned. >> >> Is there something else I need to do before calling this function? >> >> Thanks, >> Ryan >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryanm@REDACTED Thu May 17 11:54:08 2018 From: ryanm@REDACTED (Ryan Maclear) Date: Thu, 17 May 2018 11:54:08 +0200 Subject: [erlang-questions] proper char() type In-Reply-To: References: Message-ID: Hi All, I've gone through the code for the integer() type and have found where my issue arises from. the proper_gen:pick/1 function has a default Size of 10, but the proper_gen:pick/2 function provides for a size to be passed. If I call: proper_gen:pick(proper_types:char(), 1114111), I get results as expected. Since the definition of char() is equivalent to integer(0,16#10ffff), I would have expected the proper_gen:pick_type(proper_types:char()) call to result in the arity 2 version of this function being called, or at least have the Size set to the upper bound of the integer range in the integer_gen function call, either in proper_types:integer_gen/2 function call. Does this make sense or am I missing something? Regards, Ryan Maclear On 17 May 2018 at 11:08, Ryan Maclear wrote: > Appologies, I hit send before finisinh the email. Here is the complete > mail: > > I'm busy learning to use proper and have the following issue. When calling > the function > > From the API docs, the char() type has a range of 0 to (1114111) 16#10ffff. > > When calling the function > > proper_gen:pick(proper_types:term()). > > manually many times (I've tried in around 3000 times) and on different > VMs, the function always seems to return only one of the following values: > > {ok,0}, > {ok,2}, > {ok,3}, > {ok,6} > {ok,12} and > {ok,13} > > I see the same behaviour for > > proper_gen:pick(proper_types:range(0,16#10ffff)). > > which I believe is equivalent. > > I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a > ubuntu trusty docker image with erlang 19.3. > > In all three vms I see the same set of results being returned. > > Is there something else I need to do before calling this function? > > Thanks, > Ryan > > > On 17 May 2018 at 11:07, Ryan Maclear wrote: > >> Good Day, >> >> I'm busy learning to use proper and have the following issue. When >> calling the function >> >> From the API docs, the char() type has a range of 0 to (1114111) >> 16#10ffff. >> >> When calling the function >> >> proper_gen:pick(proper_types:term()). >> >> manually many times (I've tried in around 3000 times) and on different >> VMs, the function always seems to return only one of the following values: >> >> {ok,0}, >> {ok,2}, >> {ok,3}, >> {ok,6} >> {ok,12} and >> {ok,13} >> >> I see the same behaviour for >> >> >> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >> ubuntu trusty docker image with erlang 19.3. >> >> In all three vms I see the same set of results being returned. >> >> Is there something else I need to do before calling this function? >> >> Thanks, >> Ryan >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryanm@REDACTED Thu May 17 12:14:10 2018 From: ryanm@REDACTED (Ryan Maclear) Date: Thu, 17 May 2018 12:14:10 +0200 Subject: [erlang-questions] proper char() type In-Reply-To: References: Message-ID: Ah ok, that makes perfect sense. So manually calling the pick/1 function with just the type will always start with the Size parameter set to 10, but over time the internals would increase this for repeated calls, and thus stretching the sample space to the max range. Thanks for clearing that up. Kind Regards, Ryan Maclear On 17 May 2018 at 12:02, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > Most QuickCheck systems have an internal size parameter which is slowly > increased over the course of testing. If you manually bump it by a lot, you > get a better range: > > 16> proper_gen:pick(proper_types:resize(8000, proper_types:integer(0, > 16#10ffff))). > {ok,5050} > > However, I feel most of propers generators are botched and don't really > check what they should. EQC Mini for instance: > > 5> eqc_gen:sample(eqc_gen:choose(0, 16#10ffff)). > 482145 > 1062500 > 211296 > 269049 > 863625 > 4346 > 441921 > 719972 > 62442 > 703915 > 944194 > ok > > which is far more the range you would expect. Also note that the pick/1 > function is kind of dangerous since it leads people down a path where their > generators have less randomness than they should have. The right solution > often involves ?LET(X, generator(), <[X] E x>) and friends. > > > > On Thu, May 17, 2018 at 11:34 AM Ryan Maclear > wrote: > >> Appologies, I hit send before finisinh the email. Here is the complete >> mail: >> >> I'm busy learning to use proper and have the following issue. When >> calling the function >> >> From the API docs, the char() type has a range of 0 to (1114111) >> 16#10ffff. >> >> When calling the function >> >> proper_gen:pick(proper_types:term()). >> >> manually many times (I've tried in around 3000 times) and on different >> VMs, the function always seems to return only one of the following values: >> >> {ok,0}, >> {ok,2}, >> {ok,3}, >> {ok,6} >> {ok,12} and >> {ok,13} >> >> I see the same behaviour for >> >> proper_gen:pick(proper_types:range(0,16#10ffff)). >> >> which I believe is equivalent. >> >> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >> ubuntu trusty docker image with erlang 19.3. >> >> In all three vms I see the same set of results being returned. >> >> Is there something else I need to do before calling this function? >> >> Thanks, >> Ryan >> >> >> On 17 May 2018 at 11:07, Ryan Maclear wrote: >> >>> Good Day, >>> >>> I'm busy learning to use proper and have the following issue. When >>> calling the function >>> >>> From the API docs, the char() type has a range of 0 to (1114111) >>> 16#10ffff. >>> >>> When calling the function >>> >>> proper_gen:pick(proper_types:term()). >>> >>> manually many times (I've tried in around 3000 times) and on different >>> VMs, the function always seems to return only one of the following values: >>> >>> {ok,0}, >>> {ok,2}, >>> {ok,3}, >>> {ok,6} >>> {ok,12} and >>> {ok,13} >>> >>> I see the same behaviour for >>> >>> >>> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >>> ubuntu trusty docker image with erlang 19.3. >>> >>> In all three vms I see the same set of results being returned. >>> >>> Is there something else I need to do before calling this function? >>> >>> Thanks, >>> Ryan >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Thu May 17 14:04:32 2018 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Thu, 17 May 2018 14:04:32 +0200 Subject: [erlang-questions] proper char() type In-Reply-To: References: Message-ID: You should also take a look at proper_gen:sample and sampleshrink ;) Also Fred Herbet?s propertesting.com! On Thu 17 May 2018 at 12:36, Ryan Maclear wrote: > Ah ok, that makes perfect sense. So manually calling the pick/1 function > with just the type will always start with the Size parameter set to 10, but > over time the internals would increase this for repeated calls, and thus > stretching the sample space to the max range. > > Thanks for clearing that up. > > Kind Regards, > Ryan Maclear > > > On 17 May 2018 at 12:02, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED> wrote: > >> Most QuickCheck systems have an internal size parameter which is slowly >> increased over the course of testing. If you manually bump it by a lot, you >> get a better range: >> >> 16> proper_gen:pick(proper_types:resize(8000, proper_types:integer(0, >> 16#10ffff))). >> {ok,5050} >> >> However, I feel most of propers generators are botched and don't really >> check what they should. EQC Mini for instance: >> >> 5> eqc_gen:sample(eqc_gen:choose(0, 16#10ffff)). >> 482145 >> 1062500 >> 211296 >> 269049 >> 863625 >> 4346 >> 441921 >> 719972 >> 62442 >> 703915 >> 944194 >> ok >> >> which is far more the range you would expect. Also note that the pick/1 >> function is kind of dangerous since it leads people down a path where their >> generators have less randomness than they should have. The right solution >> often involves ?LET(X, generator(), <[X] E x>) and friends. >> >> >> >> On Thu, May 17, 2018 at 11:34 AM Ryan Maclear >> wrote: >> >>> Appologies, I hit send before finisinh the email. Here is the complete >>> mail: >>> >>> I'm busy learning to use proper and have the following issue. When >>> calling the function >>> >>> From the API docs, the char() type has a range of 0 to (1114111) >>> 16#10ffff. >>> >>> When calling the function >>> >>> proper_gen:pick(proper_types:term()). >>> >>> manually many times (I've tried in around 3000 times) and on different >>> VMs, the function always seems to return only one of the following values: >>> >>> {ok,0}, >>> {ok,2}, >>> {ok,3}, >>> {ok,6} >>> {ok,12} and >>> {ok,13} >>> >>> I see the same behaviour for >>> >>> proper_gen:pick(proper_types:range(0,16#10ffff)). >>> >>> which I believe is equivalent. >>> >>> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >>> ubuntu trusty docker image with erlang 19.3. >>> >>> In all three vms I see the same set of results being returned. >>> >>> Is there something else I need to do before calling this function? >>> >>> Thanks, >>> Ryan >>> >>> >>> On 17 May 2018 at 11:07, Ryan Maclear wrote: >>> >>>> Good Day, >>>> >>>> I'm busy learning to use proper and have the following issue. When >>>> calling the function >>>> >>>> From the API docs, the char() type has a range of 0 to (1114111) >>>> 16#10ffff. >>>> >>>> When calling the function >>>> >>>> proper_gen:pick(proper_types:term()). >>>> >>>> manually many times (I've tried in around 3000 times) and on different >>>> VMs, the function always seems to return only one of the following values: >>>> >>>> {ok,0}, >>>> {ok,2}, >>>> {ok,3}, >>>> {ok,6} >>>> {ok,12} and >>>> {ok,13} >>>> >>>> I see the same behaviour for >>>> >>>> >>>> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >>>> ubuntu trusty docker image with erlang 19.3. >>>> >>>> In all three vms I see the same set of results being returned. >>>> >>>> Is there something else I need to do before calling this function? >>>> >>>> Thanks, >>>> Ryan >>>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Cheers, -- Pierre Fenoll -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu May 17 14:05:06 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 17 May 2018 14:05:06 +0200 Subject: [erlang-questions] proper char() type In-Reply-To: References: Message-ID: I don't think you are missing anything. Some QuickCheck systems will use a size parameter like 1-100 and then they "scale" the range over that size. If you ask for, say, 30 test cases, they usually scale 1-100 over the first 15, and then they run 15 with the full range. The idea, roughly, is that while you eventually want to check a full range, but many errors occur when a range is small. And small cases are often easier to debug. This notion goes hand in hand with shrinking: if you _do_ find a large test case, pick a strategy for finding a failure, but for a simpler generated input. As for integers, I often use something like: %% @doc pow_2_int/0 generates integers close to a power of two %% It turns out that integers around powers of two are often able to mess up stuff %% because of their bitwise representation. This generator generates integers close %% to a power of two deliberately. %% @end pow_2_int() -> ?LET({Sign, Exponent, Perturb}, {sign(), choose(0, 128), choose(-3, 3)}, Sign * pow(2, Exponent) + Perturb). sign() -> elements([1, -1]). pow(0, 0) -> 0; pow(_Base, 0) -> 1; pow(Base, N) -> Base * pow(Base, N-1). (This is from [0]) It doesn't generate any integer, but since a lot of systems cannot handle corner case integers well, this is an excellent way to check them. Especially if they have two-complement representations and have one more negative value than positive. The above code sequence has found bugs in Erlangs bignum handling btw. [0] https://github.com/jlouis/eqc_lib/blob/master/eqc_lib.erl#L14-L27 On Thu, May 17, 2018 at 12:35 PM Ryan Maclear wrote: > Hi All, > > I've gone through the code for the integer() type and have found where my > issue arises from. > > the proper_gen:pick/1 function has a default Size of 10, but the > proper_gen:pick/2 function provides for a size to be passed. > > If I call: > > proper_gen:pick(proper_types:char(), 1114111), I get results as expected. > > Since the definition of char() is equivalent to integer(0,16#10ffff), I > would have expected the proper_gen:pick_type(proper_types:char()) call to > result in the arity 2 version of this function being called, or at least > have the Size set to the upper bound of the integer range in the > integer_gen function call, either in proper_types:integer_gen/2 function > call. > > Does this make sense or am I missing something? > > Regards, > Ryan Maclear > > > > > On 17 May 2018 at 11:08, Ryan Maclear wrote: > >> Appologies, I hit send before finisinh the email. Here is the complete >> mail: >> >> I'm busy learning to use proper and have the following issue. When >> calling the function >> >> From the API docs, the char() type has a range of 0 to (1114111) >> 16#10ffff. >> >> When calling the function >> >> proper_gen:pick(proper_types:term()). >> >> manually many times (I've tried in around 3000 times) and on different >> VMs, the function always seems to return only one of the following values: >> >> {ok,0}, >> {ok,2}, >> {ok,3}, >> {ok,6} >> {ok,12} and >> {ok,13} >> >> I see the same behaviour for >> >> proper_gen:pick(proper_types:range(0,16#10ffff)). >> >> which I believe is equivalent. >> >> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >> ubuntu trusty docker image with erlang 19.3. >> >> In all three vms I see the same set of results being returned. >> >> Is there something else I need to do before calling this function? >> >> Thanks, >> Ryan >> >> >> On 17 May 2018 at 11:07, Ryan Maclear wrote: >> >>> Good Day, >>> >>> I'm busy learning to use proper and have the following issue. When >>> calling the function >>> >>> From the API docs, the char() type has a range of 0 to (1114111) >>> 16#10ffff. >>> >>> When calling the function >>> >>> proper_gen:pick(proper_types:term()). >>> >>> manually many times (I've tried in around 3000 times) and on different >>> VMs, the function always seems to return only one of the following values: >>> >>> {ok,0}, >>> {ok,2}, >>> {ok,3}, >>> {ok,6} >>> {ok,12} and >>> {ok,13} >>> >>> I see the same behaviour for >>> >>> >>> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >>> ubuntu trusty docker image with erlang 19.3. >>> >>> In all three vms I see the same set of results being returned. >>> >>> Is there something else I need to do before calling this function? >>> >>> Thanks, >>> Ryan >>> >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryanm@REDACTED Thu May 17 14:16:06 2018 From: ryanm@REDACTED (Ryan Maclear) Date: Thu, 17 May 2018 14:16:06 +0200 Subject: [erlang-questions] proper char() type In-Reply-To: References: Message-ID: Yip - I'm working through propertesting.com again, but paying more attention and looking at it a lot closer this time as I have a real need for doing some stateful property testing. It really is a great resource. On 17 May 2018 at 14:04, Pierre Fenoll wrote: > You should also take a look at > proper_gen:sample and sampleshrink ;) > > Also Fred Herbet?s propertesting.com! > > On Thu 17 May 2018 at 12:36, Ryan Maclear wrote: > >> Ah ok, that makes perfect sense. So manually calling the pick/1 function >> with just the type will always start with the Size parameter set to 10, but >> over time the internals would increase this for repeated calls, and thus >> stretching the sample space to the max range. >> >> Thanks for clearing that up. >> >> Kind Regards, >> Ryan Maclear >> >> >> On 17 May 2018 at 12:02, Jesper Louis Andersen < >> jesper.louis.andersen@REDACTED> wrote: >> >>> Most QuickCheck systems have an internal size parameter which is slowly >>> increased over the course of testing. If you manually bump it by a lot, you >>> get a better range: >>> >>> 16> proper_gen:pick(proper_types:resize(8000, proper_types:integer(0, >>> 16#10ffff))). >>> {ok,5050} >>> >>> However, I feel most of propers generators are botched and don't really >>> check what they should. EQC Mini for instance: >>> >>> 5> eqc_gen:sample(eqc_gen:choose(0, 16#10ffff)). >>> 482145 >>> 1062500 >>> 211296 >>> 269049 >>> 863625 >>> 4346 >>> 441921 >>> 719972 >>> 62442 >>> 703915 >>> 944194 >>> ok >>> >>> which is far more the range you would expect. Also note that the pick/1 >>> function is kind of dangerous since it leads people down a path where their >>> generators have less randomness than they should have. The right solution >>> often involves ?LET(X, generator(), <[X] E x>) and friends. >>> >>> >>> >>> On Thu, May 17, 2018 at 11:34 AM Ryan Maclear >>> wrote: >>> >>>> Appologies, I hit send before finisinh the email. Here is the complete >>>> mail: >>>> >>>> I'm busy learning to use proper and have the following issue. When >>>> calling the function >>>> >>>> From the API docs, the char() type has a range of 0 to (1114111) >>>> 16#10ffff. >>>> >>>> When calling the function >>>> >>>> proper_gen:pick(proper_types:term()). >>>> >>>> manually many times (I've tried in around 3000 times) and on different >>>> VMs, the function always seems to return only one of the following values: >>>> >>>> {ok,0}, >>>> {ok,2}, >>>> {ok,3}, >>>> {ok,6} >>>> {ok,12} and >>>> {ok,13} >>>> >>>> I see the same behaviour for >>>> >>>> proper_gen:pick(proper_types:range(0,16#10ffff)). >>>> >>>> which I believe is equivalent. >>>> >>>> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on a >>>> ubuntu trusty docker image with erlang 19.3. >>>> >>>> In all three vms I see the same set of results being returned. >>>> >>>> Is there something else I need to do before calling this function? >>>> >>>> Thanks, >>>> Ryan >>>> >>>> >>>> On 17 May 2018 at 11:07, Ryan Maclear wrote: >>>> >>>>> Good Day, >>>>> >>>>> I'm busy learning to use proper and have the following issue. When >>>>> calling the function >>>>> >>>>> From the API docs, the char() type has a range of 0 to (1114111) >>>>> 16#10ffff. >>>>> >>>>> When calling the function >>>>> >>>>> proper_gen:pick(proper_types:term()). >>>>> >>>>> manually many times (I've tried in around 3000 times) and on different >>>>> VMs, the function always seems to return only one of the following values: >>>>> >>>>> {ok,0}, >>>>> {ok,2}, >>>>> {ok,3}, >>>>> {ok,6} >>>>> {ok,12} and >>>>> {ok,13} >>>>> >>>>> I see the same behaviour for >>>>> >>>>> >>>>> I have tried on erlang 19.3.6.9, erlang 20.3.6 on a Mac as well as on >>>>> a ubuntu trusty docker image with erlang 19.3. >>>>> >>>>> In all three vms I see the same set of results being returned. >>>>> >>>>> Is there something else I need to do before calling this function? >>>>> >>>>> Thanks, >>>>> Ryan >>>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- > > Cheers, > -- > Pierre Fenoll > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri May 18 08:14:45 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 18 May 2018 09:14:45 +0300 Subject: [erlang-questions] webrtc lib In-Reply-To: References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: Hi. We have webrtc implementation inside Flussonic. It is a pure full webrtc implementation, not websocket signalling, but full video/audio transfer. You are speaking about it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri May 18 08:15:31 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 18 May 2018 09:15:31 +0300 Subject: [erlang-questions] how to start and stop erlang daemon In-Reply-To: References: Message-ID: How can you signal to system administrator that config is invalid? -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Fri May 18 09:28:20 2018 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 18 May 2018 08:28:20 +0100 Subject: [erlang-questions] how to start and stop erlang daemon In-Reply-To: References: Message-ID: On 18 May 2018 at 07:15, Max Lapshin wrote: > How can you signal to system administrator that config is invalid? We do a mixture of the following: - Erlang node refuses to start; upstart (or systemd) or monit send an error email and attempt to restart it. - Erlang node reports an error; lager backend turns that into an error email. - Erlang node fails to do whatever it's supposed to do; external check notices (nagios can't connect to port, end-to-end smoke test fails, e.g.), notifies administrator. System administrator sees the spike in the error emails, takes a look, fixes the problem. From hellkvist@REDACTED Fri May 18 10:22:30 2018 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Fri, 18 May 2018 10:22:30 +0200 Subject: [erlang-questions] webrtc lib In-Reply-To: References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: > 18 maj 2018 kl. 08:14 skrev Max Lapshin : > > Hi. > > We have webrtc implementation inside Flussonic. > > It is a pure full webrtc implementation, not websocket signalling, but full video/audio transfer. > > You are speaking about it? That is interesting. We were actually talking about integrating Janus (https://janus.conf.meetecho.com) with Erlang but it is interesting to hear about an Erlang webrtc implementation. What is exactly that you have implemented? Janus does srtp relaying and does ice negotiation with STUN/TURN as well as allowing you to write plugins with more specific application logic. What type of webrtc support exists in Flussonic? Another late development is the added support for webrtc in latest Gstreamer (which also uses libnice ). I?m thinking that, if one were to wrap webrtc into and Erlang environment that perhaps the Gstreamer pipeline is a more flexible integration point than Janus? I have tried the new Gstreamer with some limited simple use cases and it seems to work fine. Stefan From bchesneau@REDACTED Fri May 18 10:43:40 2018 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 18 May 2018 10:43:40 +0200 Subject: [erlang-questions] webrtc lib In-Reply-To: References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: Hi Max, I knew you had an implementation but it doesn't seem you opensourced it yet, right? At least I didn't found it. If you do that would be really awesome! If you need any help for it, feel free to bug me :) I mainly need to have data channels, so if they are missing I could start by adding them. Far better than reinventing the wheel :) - benoit On Fri, May 18, 2018 at 8:14 AM, Max Lapshin wrote: > Hi. > > We have webrtc implementation inside Flussonic. > > It is a pure full webrtc implementation, not websocket signalling, but > full video/audio transfer. > > You are speaking about it? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Fri May 18 16:38:19 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 18 May 2018 16:38:19 +0200 Subject: [erlang-questions] Factorization challenge Message-ID: <20180518143819.GA12880@erix.ericsson.se> Hi all! For an improved long period PRNG it is needed to factor a certain number into prime numbers. The number is: 2^928 - 1 So if anyone has got the resources to do that I would be grateful! Good luck to any number crunchers out there! -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From vladdu55@REDACTED Fri May 18 19:15:00 2018 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Fri, 18 May 2018 19:15:00 +0200 Subject: [erlang-questions] Factorization challenge In-Reply-To: <20180518143819.GA12880@erix.ericsson.se> References: <20180518143819.GA12880@erix.ericsson.se> Message-ID: - 2269 007733 883335 972287 082669 296112 915239 349672 942191 252221 331572 442536 403137 824056 312817 862695 551072 066953 619064 625508 194663 368599 769448 406663 254670 871573 830845 597595 897613 333042 429214 224697 474472 410882 236254 024057 110212 260250 671521 235807 709272 244389 361641 091086 035023 229622 419455 (280 digits) = 3 ? 5 ? 17 ? 59 ? 233 ? 257 ? 929 ? 1103 ? 2089 ? 5569 ? 8353 ? 59393 ? 65537 ? 3 033169 ? 39 594977 ? 107 367629 ? 536 903681 ? 748 264961 ? 2245 984577 ? 239 686663 718401 ? 15 929619 591127 520827 829953 ? 82280 195167 144119 832390 568177 ? 6033 312171 721035 031651 315652 130497 (34 digits) ? 18 774318 450142 955120 650303 957350 521748 903233 (44 digits) ? 15 694604 006012 505869 851221 169365 594050 637743 819041 (50 digits) (unverified, as returned by https://www.alpertron.com.ar/ECM.HTM) regards, Vlad On Fri, May 18, 2018 at 4:40 PM Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > Hi all! > > For an improved long period PRNG it is needed to factor a certain number > into prime numbers. > > The number is: 2^928 - 1 > > So if anyone has got the resources to do that I would be grateful! > > Good luck to any number crunchers out there! > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sat May 19 16:47:14 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 19 May 2018 17:47:14 +0300 Subject: [erlang-questions] webrtc lib In-Reply-To: References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: I'm completely against writing any wrappers around C network libraries, so we have pure erlang implementation that is a peer in p2p webrtc connection. We can accept published video or stream video to network, handling stun, turn, srtp, rtp, rtcp and so on including unpacking frames and packing them back. We do not have data channels. Benoit, I will get in touch with you and discuss it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat May 19 18:01:36 2018 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 19 May 2018 18:01:36 +0200 Subject: [erlang-questions] webrtc lib In-Reply-To: References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: On Sat, May 19, 2018 at 4:47 PM, Max Lapshin wrote: > I'm completely against writing any wrappers around C network libraries, so > we have pure erlang implementation that is a peer in p2p webrtc connection. > I concur ... > > We can accept published video or stream video to network, handling stun, > turn, srtp, rtp, rtcp and so on including unpacking frames and packing them > back. > > We do not have data channels. Benoit, I will get in touch with you and > discuss it. > Thanks! Feel free to drop me a mail or contact me on irc any time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gugugugu2006@REDACTED Sun May 20 13:40:01 2018 From: gugugugu2006@REDACTED (gugu Gugug) Date: Sun, 20 May 2018 07:40:01 -0400 Subject: [erlang-questions] (no subject) Message-ID: <1526816404.KMgqf3dEbYKIkKMh2fFv1L@mf-smf-ucb023c3> http://explore.obduae.com Gugu Gugug -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sun May 20 21:38:08 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 20 May 2018 22:38:08 +0300 Subject: [erlang-questions] webrtc lib In-Reply-To: References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: There is an obvious problem: right now webrtc implementation is deeply merged with flussonic internals like: websocket_info({dtls, key_material, #srtp_params{} = SrtpParams}, Req, #publish{stream_name = StreamName} = State) -> % events:debug("Got SRTP params: ~p", [SrtpParams]), % {ok, DTS} = live_stream:getattr(StreamName, last_dts), % Now = erlang:system_time(milli_seconds), MI = live_stream:media_info(StreamName), live_stream is a internal module and not supposed to be exported to public-available library. It will take some time to extract. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Mon May 21 11:36:57 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 21 May 2018 11:36:57 +0200 Subject: [erlang-questions] Factorization challenge In-Reply-To: References: <20180518143819.GA12880@erix.ericsson.se> Message-ID: <20180521093657.GB6049@erix.ericsson.se> On Fri, May 18, 2018 at 07:15:00PM +0200, Vlad Dumitrescu wrote: > - 2269 007733 883335 972287 082669 296112 915239 349672 942191 252221 > 331572 442536 403137 824056 312817 862695 551072 066953 619064 625508 > 194663 368599 769448 406663 254670 871573 830845 597595 897613 333042 > 429214 224697 474472 410882 236254 024057 110212 260250 671521 235807 > 709272 244389 361641 091086 035023 229622 419455 (280 digits) = 3 ? 5 ? 17 > ? 59 ? 233 ? 257 ? 929 ? 1103 ? 2089 ? 5569 ? 8353 ? 59393 ? 65537 ? 3 > 033169 ? 39 594977 ? 107 367629 ? 536 903681 ? 748 264961 ? 2245 984577 ? > 239 686663 718401 ? 15 929619 591127 520827 829953 ? 82280 195167 144119 > 832390 568177 ? 6033 312171 721035 031651 315652 130497 (34 digits) ? 18 > 774318 450142 955120 650303 957350 521748 903233 (44 digits) ? 15 694604 > 006012 505869 851221 169365 594050 637743 819041 (50 digits) > > (unverified, as returned by https://www.alpertron.com.ar/ECM.HTM) Amazing! 1> (1 bsl 928) - 1 =:= 3 * 5 * 17 * 59 * 233 * 257 * 929 * 1103 * 2089 * 5569 * 8353 * 59393 * 65537 * 3033169 * 39594977 * 107367629 * 536903681 * 748264961 * 2245984577 * 239686663718401 * 15929619591127520827829953 * 82280195167144119832390568177 * 6033312171721035031651315652130497 * 18774318450142955120650303957350521748903233 * 15694604006012505869851221169365594050637743819041. => true The numbers add up! (multiply up) And WolframAlpha confirms all factors to be prime numbers. Excellent! Thank you ! / Raimo > > regards, > Vlad > > > On Fri, May 18, 2018 at 4:40 PM Raimo Niskanen < > raimo+erlang-questions@REDACTED> wrote: > > > Hi all! > > > > For an improved long period PRNG it is needed to factor a certain number > > into prime numbers. > > > > The number is: 2^928 - 1 > > > > So if anyone has got the resources to do that I would be grateful! > > > > Good luck to any number crunchers out there! > > -- > > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From peer@REDACTED Tue May 22 13:46:01 2018 From: peer@REDACTED (Peer Stritzinger) Date: Tue, 22 May 2018 13:46:01 +0200 Subject: [erlang-questions] Free GRiSP tutorial at CodeBEAM Stockholm next Wednesday, looking for more attendees! Message-ID: <0D51A193-4D11-44A7-95A8-80AC1985DDE1@stritzinger.com> We have a GRiSP embedded system tutorial at the CodeBEAM in Stockholm. Since it was not added to the conference website until long after our submission, we are still looking for a few more attendees! BTW: Tutorial tickets are free and don?t require a ticket to the conference. So ideal for local students Description: http://codebeam.io/conferences/code-beam-sto-2018/training Scroll down to: GRISP TRAINING - IOT LAB WITH GRISP, BARE METAL ERLANG, SENSORS AND ACTUATORS (30 MAY) - PEER STRITZINGER AND ADAM LINDBERG -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3866 bytes Desc: not available URL: From z@REDACTED Wed May 23 17:28:55 2018 From: z@REDACTED (Danil Zagoskin) Date: Wed, 23 May 2018 18:28:55 +0300 Subject: [erlang-questions] UDP receive performance Message-ID: Hi! We have a performance problem receiving lots of UDP traffic. There are a lot (about 70) of UDP receive processes, each handling about 1 to 10 megabits of multicast traffic, with {active, N}. msacc summary on my OSX laptop, build from OTP master c30309e799212b080c39ee2f91af3f9a0383d767 (Apr 19): Thread alloc aux bifbusy_wait check_io emulator ets gc gc_full nif other port send sleep timers scheduler 30.02% 0.92% 2.86% 24.66% 0.01% 9.61% 0.03% 1.25% 0.20% 0.13% 2.34% 9.33% 0.41% 17.78% 0.44% Linux production server behaves the same way (we do not have extended msacc there yet, so most of alloc goes to port). perf top (on Linux production) says there's a lot of unaligned memmove: 69.76% libc-2.24.so [.] __memmove_sse2_unaligned_erms 6.13% beam.smp [.] process_main 2.02% beam.smp [.] erts_schedule 0.87% [kernel] [k] copy_user_enhanced_fast_string I'll try to make a minimal example for this. Maybe there are simple recommendations on optimizing this kind of load? -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Thu May 24 09:23:57 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 24 May 2018 09:23:57 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: Message-ID: <20180524072357.GA97377@erix.ericsson.se> On Wed, May 23, 2018 at 06:28:55PM +0300, Danil Zagoskin wrote: > Hi! > > We have a performance problem receiving lots of UDP traffic. > There are a lot (about 70) of UDP receive processes, each handling about 1 > to 10 megabits of multicast traffic, with {active, N}. Whenever someone has UDP receive performance problems one has to ask if you have seen the Erlang socket option {read_packets,N}? See http://erlang.org/doc/man/inet.html#setopts-2 > > msacc summary on my OSX laptop, build from OTP master > c30309e799212b080c39ee2f91af3f9a0383d767 (Apr 19): > > > Thread alloc aux bifbusy_wait check_io emulator > ets gc gc_full nif other port send sleep > timers > scheduler 30.02% 0.92% 2.86% 24.66% 0.01% 9.61% > 0.03% 1.25% 0.20% 0.13% 2.34% 9.33% 0.41% 17.78% > 0.44% > > > Linux production server behaves the same way (we do not have extended msacc > there yet, so most of alloc goes to port). > > perf top (on Linux production) says there's a lot of unaligned memmove: > > 69.76% libc-2.24.so [.] __memmove_sse2_unaligned_erms > 6.13% beam.smp [.] process_main > 2.02% beam.smp [.] erts_schedule > 0.87% [kernel] [k] copy_user_enhanced_fast_string > > > I'll try to make a minimal example for this. > Maybe there are simple recommendations on optimizing this kind of load? > > -- > Danil Zagoskin | z@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From z@REDACTED Thu May 24 12:03:35 2018 From: z@REDACTED (Danil Zagoskin) Date: Thu, 24 May 2018 13:03:35 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: <20180524072357.GA97377@erix.ericsson.se> References: <20180524072357.GA97377@erix.ericsson.se> Message-ID: Yes, we have {read_packets, 100} in receive socket options. On Thu, May 24, 2018 at 10:23 AM, Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > On Wed, May 23, 2018 at 06:28:55PM +0300, Danil Zagoskin wrote: > > Hi! > > > > We have a performance problem receiving lots of UDP traffic. > > There are a lot (about 70) of UDP receive processes, each handling about > 1 > > to 10 megabits of multicast traffic, with {active, N}. > > Whenever someone has UDP receive performance problems one has to ask if you > have seen the Erlang socket option {read_packets,N}? > > See http://erlang.org/doc/man/inet.html#setopts-2 > > > > > msacc summary on my OSX laptop, build from OTP master > > c30309e799212b080c39ee2f91af3f9a0383d767 (Apr 19): > > > > > > Thread alloc aux bifbusy_wait check_io emulator > > ets gc gc_full nif other port send sleep > > timers > > scheduler 30.02% 0.92% 2.86% 24.66% 0.01% 9.61% > > 0.03% 1.25% 0.20% 0.13% 2.34% 9.33% 0.41% 17.78% > > 0.44% > > > > > > Linux production server behaves the same way (we do not have extended > msacc > > there yet, so most of alloc goes to port). > > > > perf top (on Linux production) says there's a lot of unaligned memmove: > > > > 69.76% libc-2.24.so [.] __memmove_sse2_unaligned_erms > > 6.13% beam.smp [.] process_main > > 2.02% beam.smp [.] erts_schedule > > 0.87% [kernel] [k] copy_user_enhanced_fast_string > > > > > > I'll try to make a minimal example for this. > > Maybe there are simple recommendations on optimizing this kind of load? > > > > -- > > Danil Zagoskin | z@REDACTED > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Thu May 24 12:17:30 2018 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Thu, 24 May 2018 12:17:30 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> Message-ID: <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> OTP-21 rc1 has enhanced IO scalability. Have you tried if it is any better? UDP performance in Erlang was never great... Regards, Sergej > On 24 May 2018, at 12:03, Danil Zagoskin wrote: > > Yes, we have {read_packets, 100} in receive socket options. > > On Thu, May 24, 2018 at 10:23 AM, Raimo Niskanen > wrote: > On Wed, May 23, 2018 at 06:28:55PM +0300, Danil Zagoskin wrote: > > Hi! > > > > We have a performance problem receiving lots of UDP traffic. > > There are a lot (about 70) of UDP receive processes, each handling about 1 > > to 10 megabits of multicast traffic, with {active, N}. > > Whenever someone has UDP receive performance problems one has to ask if you > have seen the Erlang socket option {read_packets,N}? > > See http://erlang.org/doc/man/inet.html#setopts-2 > > > > > msacc summary on my OSX laptop, build from OTP master > > c30309e799212b080c39ee2f91af3f9a0383d767 (Apr 19): > > > > > > Thread alloc aux bifbusy_wait check_io emulator > > ets gc gc_full nif other port send sleep > > timers > > scheduler 30.02% 0.92% 2.86% 24.66% 0.01% 9.61% > > 0.03% 1.25% 0.20% 0.13% 2.34% 9.33% 0.41% 17.78% > > 0.44% > > > > > > Linux production server behaves the same way (we do not have extended msacc > > there yet, so most of alloc goes to port). > > > > perf top (on Linux production) says there's a lot of unaligned memmove: > > > > 69.76% libc-2.24.so [.] __memmove_sse2_unaligned_erms > > 6.13% beam.smp [.] process_main > > 2.02% beam.smp [.] erts_schedule > > 0.87% [kernel] [k] copy_user_enhanced_fast_string > > > > > > I'll try to make a minimal example for this. > > Maybe there are simple recommendations on optimizing this kind of load? > > > > -- > > Danil Zagoskin | z@REDACTED > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Danil Zagoskin | z@REDACTED _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Thu May 24 12:21:23 2018 From: z@REDACTED (Danil Zagoskin) Date: Thu, 24 May 2018 13:21:23 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: Yes, I've built a fresh master today (Erlang/OTP 21 [RELEASE CANDIDATE 1] [erts-9.3.1]), and nothing has changed. On Thu, May 24, 2018 at 1:17 PM, Sergej Jure?ko wrote: > OTP-21 rc1 has enhanced IO scalability. Have you tried if it is any > better? UDP performance in Erlang was never great... > > Regards, > Sergej > > > On 24 May 2018, at 12:03, Danil Zagoskin wrote: > > Yes, we have {read_packets, 100} in receive socket options. > > On Thu, May 24, 2018 at 10:23 AM, Raimo Niskanen < > raimo+erlang-questions@REDACTED> wrote: > >> On Wed, May 23, 2018 at 06:28:55PM +0300, Danil Zagoskin wrote: >> > Hi! >> > >> > We have a performance problem receiving lots of UDP traffic. >> > There are a lot (about 70) of UDP receive processes, each handling >> about 1 >> > to 10 megabits of multicast traffic, with {active, N}. >> >> Whenever someone has UDP receive performance problems one has to ask if >> you >> have seen the Erlang socket option {read_packets,N}? >> >> See http://erlang.org/doc/man/inet.html#setopts-2 >> >> > >> > msacc summary on my OSX laptop, build from OTP master >> > c30309e799212b080c39ee2f91af3f9a0383d767 (Apr 19): >> > >> > >> > Thread alloc aux bifbusy_wait check_io emulator >> > ets gc gc_full nif other port send sleep >> > timers >> > scheduler 30.02% 0.92% 2.86% 24.66% 0.01% 9.61% >> > 0.03% 1.25% 0.20% 0.13% 2.34% 9.33% 0.41% 17.78% >> > 0.44% >> > >> > >> > Linux production server behaves the same way (we do not have extended >> msacc >> > there yet, so most of alloc goes to port). >> > >> > perf top (on Linux production) says there's a lot of unaligned memmove: >> > >> > 69.76% libc-2.24.so [.] __memmove_sse2_unaligned_erms >> > 6.13% beam.smp [.] process_main >> > 2.02% beam.smp [.] erts_schedule >> > 0.87% [kernel] [k] copy_user_enhanced_fast_string >> > >> > >> > I'll try to make a minimal example for this. >> > Maybe there are simple recommendations on optimizing this kind of load? >> > >> > -- >> > Danil Zagoskin | z@REDACTED >> >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> >> -- >> >> / Raimo Niskanen, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Danil Zagoskin | z@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Thu May 24 12:24:37 2018 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 24 May 2018 12:24:37 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: Can you run perf with "--call-graph dwarf" and see which functions it is that call memmove? On Thu, May 24, 2018 at 12:21 PM, Danil Zagoskin wrote: > Yes, I've built a fresh master today (Erlang/OTP 21 [RELEASE CANDIDATE 1] > [erts-9.3.1]), and nothing has changed. > > On Thu, May 24, 2018 at 1:17 PM, Sergej Jure?ko > wrote: > >> OTP-21 rc1 has enhanced IO scalability. Have you tried if it is any >> better? UDP performance in Erlang was never great... >> >> Regards, >> Sergej >> >> >> On 24 May 2018, at 12:03, Danil Zagoskin wrote: >> >> Yes, we have {read_packets, 100} in receive socket options. >> >> On Thu, May 24, 2018 at 10:23 AM, Raimo Niskanen < >> raimo+erlang-questions@REDACTED> wrote: >> >>> On Wed, May 23, 2018 at 06:28:55PM +0300, Danil Zagoskin wrote: >>> > Hi! >>> > >>> > We have a performance problem receiving lots of UDP traffic. >>> > There are a lot (about 70) of UDP receive processes, each handling >>> about 1 >>> > to 10 megabits of multicast traffic, with {active, N}. >>> >>> Whenever someone has UDP receive performance problems one has to ask if >>> you >>> have seen the Erlang socket option {read_packets,N}? >>> >>> See http://erlang.org/doc/man/inet.html#setopts-2 >>> >>> > >>> > msacc summary on my OSX laptop, build from OTP master >>> > c30309e799212b080c39ee2f91af3f9a0383d767 (Apr 19): >>> > >>> > >>> > Thread alloc aux bifbusy_wait check_io emulator >>> > ets gc gc_full nif other port send sleep >>> > timers >>> > scheduler 30.02% 0.92% 2.86% 24.66% 0.01% 9.61% >>> > 0.03% 1.25% 0.20% 0.13% 2.34% 9.33% 0.41% 17.78% >>> > 0.44% >>> > >>> > >>> > Linux production server behaves the same way (we do not have extended >>> msacc >>> > there yet, so most of alloc goes to port). >>> > >>> > perf top (on Linux production) says there's a lot of unaligned memmove: >>> > >>> > 69.76% libc-2.24.so [.] __memmove_sse2_unaligned_erms >>> > 6.13% beam.smp [.] process_main >>> > 2.02% beam.smp [.] erts_schedule >>> > 0.87% [kernel] [k] copy_user_enhanced_fast_string >>> > >>> > >>> > I'll try to make a minimal example for this. >>> > Maybe there are simple recommendations on optimizing this kind of load? >>> > >>> > -- >>> > Danil Zagoskin | z@REDACTED >>> >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> -- >>> >>> / Raimo Niskanen, Erlang/OTP, Ericsson AB >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> -- >> Danil Zagoskin | z@REDACTED >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > > -- > Danil Zagoskin | z@REDACTED > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu May 24 12:58:54 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 24 May 2018 12:58:54 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: Message-ID: On Wed, May 23, 2018 at 5:29 PM Danil Zagoskin wrote: > Hi! > > We have a performance problem receiving lots of UDP traffic. > There are a lot (about 70) of UDP receive processes, each handling about 1 > to 10 megabits of multicast traffic, with {active, N}. > > Suppose you read the packets, and then throw everything away, as a test. Are you then fast enough, or do you have a problem still? Chances are that the problem isn't the reception by itself. memmove just means you are moving memory around a lot, but the question is: why? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu May 24 13:23:06 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 24 May 2018 13:23:06 +0200 Subject: [erlang-questions] Deployment rebar3 In-Reply-To: References: <87a2d6fe-8f70-4166-9866-dc28f9b31df6@Spark> Message-ID: On Mon, May 7, 2018 at 4:44 PM Guilherme Andrade wrote: > On 7 May 2018 at 15:03, Stefan Hellkvist wrote: > >> What's "best" is very much depending on who you ask, but, for starters >> you may consider wrapping your configuration with something that allows you >> to override config variables with environment variables (see example below). >> > > rebar3 releases provide this out of the box: > > https://www.rebar3.org/docs/releases#dynamic-configuration > > > When it works :) If you use it, copy `sys.config` to `sys.config.orig` as well and deliver this to the machine. That way, the script won't try to save `sys.config.orig` if it is not present. This makes your dynamic configuration more static and can avoid some situations where the expanded `sys.config` is being copied into `sys.config.orig`. This then removes the dynamic configuration altogether. -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Thu May 24 13:27:43 2018 From: z@REDACTED (Danil Zagoskin) Date: Thu, 24 May 2018 14:27:43 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: Message-ID: I've made a simple example. The code is at https://gist.github.com/stolen/40eebd6225faf821153f1eeb5374f068 I added the 239.0.0.0/8 route as direct loopback (lo0 in OSX) to avoid network driver overhead. 40 dummy readers are enough to eat almost all 4 cores on my macbook (quite old i7 2 GHz): Erlang/OTP 21 [RELEASE CANDIDATE 1] [erts-9.3.1] [source-dfc935298b] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] 1> c(udptest). {ok,udptest} 2> udptest:start_sender({239,9,9,9}, 3999). <0.82.0> 3> [udptest:start_reader({239,9,9,9}, 3999) || _ <- lists:seq(1, 40)]. [<0.84.0>,<0.85.0>,<0.86.0>,<0.87.0>,<0.88.0>,<0.89.0>, <0.90.0>,<0.91.0>,<0.92.0>,<0.93.0>,<0.94.0>,<0.95.0>, <0.96.0>,<0.97.0>,<0.98.0>,<0.99.0>,<0.100.0>,<0.101.0>, <0.102.0>,<0.103.0>,<0.104.0>,<0.105.0>,<0.106.0>,<0.107.0>, <0.108.0>,<0.109.0>,<0.110.0>,<0.111.0>,<0.112.0>|...] 4> msacc:start(10000), msacc:print(). ... Thread alloc aux bifbusy_wait check_io emulator ets gc gc_full nif other port send sleep timers ... scheduler 59.95% 0.62% 0.14% 10.65% 0.00% 1.40% 0.00% 0.39% 0.00% 0.00% 1.74% 17.43% 0.00% 7.62% 0.06% I'll build the fresh OTP on Linux box and check perf again. On Thu, May 24, 2018 at 1:58 PM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > On Wed, May 23, 2018 at 5:29 PM Danil Zagoskin wrote: > >> Hi! >> >> We have a performance problem receiving lots of UDP traffic. >> There are a lot (about 70) of UDP receive processes, each handling about >> 1 to 10 megabits of multicast traffic, with {active, N}. >> >> > > Suppose you read the packets, and then throw everything away, as a test. > Are you then fast enough, or do you have a problem still? Chances are that > the problem isn't the reception by itself. > > memmove just means you are moving memory around a lot, but the question > is: why? > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu May 24 13:51:32 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 24 May 2018 14:51:32 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: Message-ID: Problem is that we need these udp packets to become one single big packet. Data flow of small 1500 byte packets is killing us =( -------------- next part -------------- An HTML attachment was scrubbed... URL: From cchalasani@REDACTED Thu May 24 14:08:17 2018 From: cchalasani@REDACTED (Chaitanya Chalasani) Date: Thu, 24 May 2018 17:38:17 +0530 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: Message-ID: We ran into the same performance problem. Now we are trying to offload the UDP packets receiver and accumulator in C using nif and pass on the complete message to Erlang. Not sure if we this will be good alternative. > On 24-May-2018, at 17:21, Max Lapshin wrote: > > Problem is that we need these udp packets to become one single big packet. Data flow of small 1500 byte packets is killing us =( -------------- next part -------------- An HTML attachment was scrubbed... URL: From lubormann@REDACTED Thu May 24 16:44:40 2018 From: lubormann@REDACTED (Luca Bormann) Date: Thu, 24 May 2018 16:44:40 +0200 Subject: [erlang-questions] bet365/soap ignores namespace-prefix on elements of type "sequence" Message-ID: Hi everyone, i am using the following erlang-package to work with some WSDL-Services. https://github.com/bet365/soap the most part is working fine! ?but my WSDL-service expect that elements of type ?sequence? (including all sub-elements) have an existing namespace-prefix in the xml. The problem is that the ?bet365/soap?-package is ignoring this, when i try to request with the generated client. Please look at my simple examples: -------------------------------------------------------------------------------------- Simple Example for what is needed): -------------------------------------------------------------------------------------- my-client * * * * * 1234* * 4* * * -------------------------------------------------------------------------------------- Simple Example for what is generated by the ?bet365/soap?-package: -------------------------------------------------------------------------------------- my-client * * *
* * 1234* * 4* *
* -------------------------------------------------------------------------------------- Can someone help? Maybe there is an option to pass to ?bet365/soap?-package when generating the client? Maybe there is a code-change needed in ?bet365/soap?-package? Many thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Thu May 24 17:34:34 2018 From: z@REDACTED (Danil Zagoskin) Date: Thu, 24 May 2018 18:34:34 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: - 2.57% 0.02% 31_scheduler beam.smp [.] process_main - 2.55% process_main - 2.43% erts_schedule - 2.26% erts_port_task_execute - 2.25% packet_inet_input.isra.31 - 2.05% driver_realloc_binary - 2.05% realloc_thr_pref 1.87% __memmove_avx_unaligned_erms That's 40-core Xeon E5-2640, so 2.5% on a single scheduler is kind of 100% Also it's Linux kernel 4.9 On a machine with kernel 4.13 and quad-core Xeon E31225 on a half of E5's load we have: - 16.11% 0.10% 1_scheduler beam.smp [.] erts_schedule - 16.01% erts_schedule - 13.62% erts_port_task_execute - 13.11% packet_inet_input.isra.31 - 11.37% driver_realloc_binary - 11.33% realloc_thr_pref - 10.50% __memcpy_avx_unaligned 5.06% __memcpy_avx_unaligned + 1.04% page_fault 0.66% do_erts_alcu_realloc.constprop.31 + 0.79% 0x108f3 0.55% driver_deliver_term 1.30% sched_spin_wait Seems like kernel version may change a lot, will run more tests. But it seems like memory operations are unaligned which could be not very efficient. On Thu, May 24, 2018 at 1:24 PM, Lukas Larsson wrote: > Can you run perf with "--call-graph dwarf" and see which functions it is > that call memmove? > > On Thu, May 24, 2018 at 12:21 PM, Danil Zagoskin wrote: > >> Yes, I've built a fresh master today (Erlang/OTP 21 [RELEASE CANDIDATE 1] >> [erts-9.3.1]), and nothing has changed. >> >> On Thu, May 24, 2018 at 1:17 PM, Sergej Jure?ko > > wrote: >> >>> OTP-21 rc1 has enhanced IO scalability. Have you tried if it is any >>> better? UDP performance in Erlang was never great... >>> >>> Regards, >>> Sergej >>> >>> >>> On 24 May 2018, at 12:03, Danil Zagoskin wrote: >>> >>> Yes, we have {read_packets, 100} in receive socket options. >>> >>> On Thu, May 24, 2018 at 10:23 AM, Raimo Niskanen < >>> raimo+erlang-questions@REDACTED> wrote: >>> >>>> On Wed, May 23, 2018 at 06:28:55PM +0300, Danil Zagoskin wrote: >>>> > Hi! >>>> > >>>> > We have a performance problem receiving lots of UDP traffic. >>>> > There are a lot (about 70) of UDP receive processes, each handling >>>> about 1 >>>> > to 10 megabits of multicast traffic, with {active, N}. >>>> >>>> Whenever someone has UDP receive performance problems one has to ask if >>>> you >>>> have seen the Erlang socket option {read_packets,N}? >>>> >>>> See http://erlang.org/doc/man/inet.html#setopts-2 >>>> >>>> > >>>> > msacc summary on my OSX laptop, build from OTP master >>>> > c30309e799212b080c39ee2f91af3f9a0383d767 (Apr 19): >>>> > >>>> > >>>> > Thread alloc aux bifbusy_wait check_io emulator >>>> > ets gc gc_full nif other port send sleep >>>> > timers >>>> > scheduler 30.02% 0.92% 2.86% 24.66% 0.01% 9.61% >>>> > 0.03% 1.25% 0.20% 0.13% 2.34% 9.33% 0.41% 17.78% >>>> > 0.44% >>>> > >>>> > >>>> > Linux production server behaves the same way (we do not have extended >>>> msacc >>>> > there yet, so most of alloc goes to port). >>>> > >>>> > perf top (on Linux production) says there's a lot of unaligned >>>> memmove: >>>> > >>>> > 69.76% libc-2.24.so [.] __memmove_sse2_unaligned_erms >>>> > 6.13% beam.smp [.] process_main >>>> > 2.02% beam.smp [.] erts_schedule >>>> > 0.87% [kernel] [k] copy_user_enhanced_fast_string >>>> > >>>> > >>>> > I'll try to make a minimal example for this. >>>> > Maybe there are simple recommendations on optimizing this kind of >>>> load? >>>> > >>>> > -- >>>> > Danil Zagoskin | z@REDACTED >>>> >>>> > _______________________________________________ >>>> > erlang-questions mailing list >>>> > erlang-questions@REDACTED >>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> -- >>>> >>>> / Raimo Niskanen, Erlang/OTP, Ericsson AB >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> >>> -- >>> Danil Zagoskin | z@REDACTED >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >> >> >> -- >> Danil Zagoskin | z@REDACTED >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.a.de.jong@REDACTED Thu May 24 19:53:36 2018 From: w.a.de.jong@REDACTED (Willem de Jong) Date: Thu, 24 May 2018 19:53:36 +0200 Subject: [erlang-questions] bet365/soap ignores namespace-prefix on elements of type "sequence" In-Reply-To: References: Message-ID: Hello Luca, It is hard to say what the code should do without knowing the WSDL. If you can share it with me I can take a look. In general it is a good idea to use a tool like SoapUI to make sure that the WSDL is correct and to see some example messages. Regards, Willem 2018-05-24 16:44 GMT+02:00 Luca Bormann : > Hi everyone, > > > > i am using the following erlang-package to work with some WSDL-Services. > > > > https://github.com/bet365/soap > > > > the most part is working fine! > > ?but my WSDL-service expect that elements of type ?sequence? (including > all sub-elements) have an existing namespace-prefix in the xml. > > The problem is that the ?bet365/soap?-package is ignoring this, when i try > to request with the generated client. > > > > Please look at my simple examples: > > > > ------------------------------------------------------------ > -------------------------- > > Simple Example for what is needed): > > ------------------------------------------------------------ > -------------------------- > > > > xmlns:urn="urn:s:v2"> > > > > > > > > my-client > > > > > > * * > > * * > > * 1234* > > * 4* > > * * > > > > > > > > > > > > > > ------------------------------------------------------------ > -------------------------- > > Simple Example for what is generated by the ?bet365/soap?-package: > > ------------------------------------------------------------ > -------------------------- > > > > xmlns:urn="urn:s:v2"> > > > > > > > > my-client > > > > > > * * > > *
* > > * 1234* > > * 4* > > *
* > > > > > > > > > > > > > > ------------------------------------------------------------ > -------------------------- > > Can someone help? > > Maybe there is an option to pass to ?bet365/soap?-package when generating > the client? > > Maybe there is a code-change needed in ?bet365/soap?-package? > > > > Many thanks! > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu May 24 23:22:17 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 24 May 2018 23:22:17 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: I looked up the unaligned stuff. There are no aligned variant, and the unaligned variant just sets up a prologue before entering the main loop where you do have alignment. So I wouldn't worry about that, but more about where the calls are being made and where the memory is copied around. On Thu, May 24, 2018 at 5:35 PM Danil Zagoskin wrote: > - 2.57% 0.02% 31_scheduler beam.smp > [.] process_main > - 2.55% process_main > - 2.43% erts_schedule > - 2.26% erts_port_task_execute > - 2.25% packet_inet_input.isra.31 > - 2.05% driver_realloc_binary > - 2.05% realloc_thr_pref > 1.87% __memmove_avx_unaligned_erms > > That's 40-core Xeon E5-2640, so 2.5% on a single scheduler is kind of 100% > Also it's Linux kernel 4.9 > > > On a machine with kernel 4.13 and quad-core Xeon E31225 on a half of E5's > load we have: > - 16.11% 0.10% 1_scheduler beam.smp [.] > erts_schedule > - 16.01% erts_schedule > - 13.62% erts_port_task_execute > - 13.11% packet_inet_input.isra.31 > - 11.37% driver_realloc_binary > - 11.33% realloc_thr_pref > - 10.50% __memcpy_avx_unaligned > 5.06% __memcpy_avx_unaligned > + 1.04% page_fault > 0.66% do_erts_alcu_realloc.constprop.31 > + 0.79% 0x108f3 > 0.55% driver_deliver_term > 1.30% sched_spin_wait > > Seems like kernel version may change a lot, will run more tests. > > But it seems like memory operations are unaligned which could be not very > efficient. > > On Thu, May 24, 2018 at 1:24 PM, Lukas Larsson wrote: > >> Can you run perf with "--call-graph dwarf" and see which functions it is >> that call memmove? >> >> On Thu, May 24, 2018 at 12:21 PM, Danil Zagoskin wrote: >> >>> Yes, I've built a fresh master today (Erlang/OTP 21 [RELEASE CANDIDATE >>> 1] [erts-9.3.1]), and nothing has changed. >>> >>> On Thu, May 24, 2018 at 1:17 PM, Sergej Jure?ko < >>> sergej.jurecko@REDACTED> wrote: >>> >>>> OTP-21 rc1 has enhanced IO scalability. Have you tried if it is any >>>> better? UDP performance in Erlang was never great... >>>> >>>> Regards, >>>> Sergej >>>> >>>> >>>> On 24 May 2018, at 12:03, Danil Zagoskin wrote: >>>> >>>> Yes, we have {read_packets, 100} in receive socket options. >>>> >>>> On Thu, May 24, 2018 at 10:23 AM, Raimo Niskanen < >>>> raimo+erlang-questions@REDACTED> wrote: >>>> >>>>> On Wed, May 23, 2018 at 06:28:55PM +0300, Danil Zagoskin wrote: >>>>> > Hi! >>>>> > >>>>> > We have a performance problem receiving lots of UDP traffic. >>>>> > There are a lot (about 70) of UDP receive processes, each handling >>>>> about 1 >>>>> > to 10 megabits of multicast traffic, with {active, N}. >>>>> >>>>> Whenever someone has UDP receive performance problems one has to ask >>>>> if you >>>>> have seen the Erlang socket option {read_packets,N}? >>>>> >>>>> See http://erlang.org/doc/man/inet.html#setopts-2 >>>>> >>>>> > >>>>> > msacc summary on my OSX laptop, build from OTP master >>>>> > c30309e799212b080c39ee2f91af3f9a0383d767 (Apr 19): >>>>> > >>>>> > >>>>> > Thread alloc aux bifbusy_wait check_io emulator >>>>> > ets gc gc_full nif other port send sleep >>>>> > timers >>>>> > scheduler 30.02% 0.92% 2.86% 24.66% 0.01% 9.61% >>>>> > 0.03% 1.25% 0.20% 0.13% 2.34% 9.33% 0.41% 17.78% >>>>> > 0.44% >>>>> > >>>>> > >>>>> > Linux production server behaves the same way (we do not have >>>>> extended msacc >>>>> > there yet, so most of alloc goes to port). >>>>> > >>>>> > perf top (on Linux production) says there's a lot of unaligned >>>>> memmove: >>>>> > >>>>> > 69.76% libc-2.24.so [.] __memmove_sse2_unaligned_erms >>>>> > 6.13% beam.smp [.] process_main >>>>> > 2.02% beam.smp [.] erts_schedule >>>>> > 0.87% [kernel] [k] copy_user_enhanced_fast_string >>>>> > >>>>> > >>>>> > I'll try to make a minimal example for this. >>>>> > Maybe there are simple recommendations on optimizing this kind of >>>>> load? >>>>> > >>>>> > -- >>>>> > Danil Zagoskin | z@REDACTED >>>>> >>>>> > _______________________________________________ >>>>> > erlang-questions mailing list >>>>> > erlang-questions@REDACTED >>>>> > http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>>> -- >>>>> >>>>> / Raimo Niskanen, Erlang/OTP, Ericsson AB >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>>> >>>> >>>> -- >>>> Danil Zagoskin | z@REDACTED >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> >>> >>> >>> -- >>> Danil Zagoskin | z@REDACTED >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > > -- > Danil Zagoskin | z@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuna.prime@REDACTED Fri May 25 10:43:12 2018 From: kuna.prime@REDACTED (Karlo Kuna) Date: Fri, 25 May 2018 10:43:12 +0200 Subject: [erlang-questions] digraph fails to add vertex Message-ID: hello in my little program i am doing successive calls to digraph:add_vertex/2 and then my program deterministically fails with: ** exception error: {badrecord,digraph} in function digraph:do_add_vertex/2 (digraph.erl, line 378) in call from sets:fold_bucket/3 (sets.erl, line 310) in call from sets:fold_seg/4 (sets.erl, line 306) in call from sets:fold_segs/4 (sets.erl, line 302) however i cannot locate any problems, and the failing call is not in any way special thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuna.prime@REDACTED Fri May 25 11:12:02 2018 From: kuna.prime@REDACTED (Karlo Kuna) Date: Fri, 25 May 2018 11:12:02 +0200 Subject: [erlang-questions] digraph fails to add vertex In-Reply-To: References: Message-ID: i'm sorry for previous mail i have found my mistake i just wish that error would be something like: {badrecord, {expects, digraph}} On Fri, May 25, 2018 at 10:43 AM, Karlo Kuna wrote: > hello > in my little program i am doing successive calls to digraph:add_vertex/2 > and then my program deterministically fails with: > ** exception error: {badrecord,digraph} > in function digraph:do_add_vertex/2 (digraph.erl, line 378) > in call from sets:fold_bucket/3 (sets.erl, line 310) > in call from sets:fold_seg/4 (sets.erl, line 306) > in call from sets:fold_segs/4 (sets.erl, line 302) > > however i cannot locate any problems, and the failing call is not in any > way special > > thanks > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri May 25 11:13:06 2018 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 25 May 2018 11:13:06 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: On Thu, May 24, 2018 at 5:34 PM, Danil Zagoskin wrote: > - 2.57% 0.02% 31_scheduler beam.smp > [.] process_main > - 2.55% process_main > - 2.43% erts_schedule > - 2.26% erts_port_task_execute > - 2.25% packet_inet_input.isra.31 > - 2.05% driver_realloc_binary > - 2.05% realloc_thr_pref > 1.87% __memmove_avx_unaligned_erms > > That's 40-core Xeon E5-2640, so 2.5% on a single scheduler is kind of 100% > Also it's Linux kernel 4.9 > > > On a machine with kernel 4.13 and quad-core Xeon E31225 on a half of E5's > load we have: > - 16.11% 0.10% 1_scheduler beam.smp [.] > erts_schedule > - 16.01% erts_schedule > - 13.62% erts_port_task_execute > - 13.11% packet_inet_input.isra.31 > - 11.37% driver_realloc_binary > - 11.33% realloc_thr_pref > - 10.50% __memcpy_avx_unaligned > 5.06% __memcpy_avx_unaligned > + 1.04% page_fault > 0.66% do_erts_alcu_realloc.constprop.31 > + 0.79% 0x108f3 > 0.55% driver_deliver_term > 1.30% sched_spin_wait > > Seems like kernel version may change a lot, will run more tests. > > But it seems like memory operations are unaligned which could be not very > efficient. > I'm not able to re-produce your benchmark, for some reason I don't get the load that you get. Anyways, I stared a bit at the code and you get a lot of realloc that move, which is not good at all. Something that caught my eye was that the recbuf is 2 MB, while the packets you receive are a lot smaller. One of the side-effects of setting a large recbuf is that the user space buffer is also increased to the same value. I don't think you want this to happen in your case. What happens if you set buffer to the MTU? Why would changing the user buffer size effect performance? Well, the udp read is done into the user-space buffer of the given size. When that size is 2 MB it is placed by erts_alloc inside a SBC (single block carrier). Then later when is it known how much data was actually received, a realloc is made of the 2 MB buffer to the size of the received data. This moves the data across the SBC border to the MBS (multi block carrier) and the realloc will have to copy the data in the binary. So by lowering the user-space buffer to be small enough to be placed in the MBC from the start, the move in realloc should disappear. That is if my theory is correct. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From gfborn@REDACTED Fri May 25 14:22:11 2018 From: gfborn@REDACTED (Grigory Fateyev) Date: Fri, 25 May 2018 15:22:11 +0300 Subject: [erlang-questions] From rebar2 to rebar3 Message-ID: Hello! We have a big project with a lot of dependencies and legacy code. I have a task to move it to rebar3. I could compile and make a release, but couldn't start it. It returns me: {error_logger,{{2018,5,25},{11,30,16}},crash_report,[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.1970.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,riak_ensemble_clock},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[kernel_sup,<0.1946.0>]},{messages,[]},{links,[<0.1947.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,124}],[]]} {error_logger,{{2018,5,25},{11,30,16}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,riak_ensemble_clock}},{offender,[{pid,undefined},{name,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} {error_logger,{{2018,5,25},{11,30,17}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.1945.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,riak_ensemble_clock}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.1944.0>]},{messages,[{'EXIT',<0.1946.0>,normal}]},{links,[<0.1944.0>,<0.1943.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,163}],[]]} I have two theories why it so, either not all deps in the release or don't start needed apps. How to understand the issue? I've read 'crash.dump', but it has a few info to get the reason. Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From taavi@REDACTED Fri May 25 14:42:17 2018 From: taavi@REDACTED (taavi@REDACTED) Date: Fri, 25 May 2018 15:42:17 +0300 Subject: [erlang-questions] From rebar2 to rebar3 In-Reply-To: References: Message-ID: <2CB89A5C-4002-454F-ABCA-45D363E9F7B0@uninet.ee> This log indicates, that failure is in ?on_load_function_fail?, which tries to load ?riak_ensemble_clock' - this is module trying to load NIF i.e. natively implemented function i.e. C-compiled module - quite likely something bad had happened and C module is not compiled correctly and is not present Google indicates, that people have struggled with similar problems before and had to help rebar configuration https://github.com/basho/riak_ensemble/issues/122 Maybe this will help you further.. best regards, taavi > On 25 May 2018, at 15:22, Grigory Fateyev wrote: > > Hello! > > We have a big project with a lot of dependencies and legacy code. I have a task to move it to rebar3. I could compile and make a release, but couldn't start it. It returns me: > > {error_logger,{{2018,5,25},{11,30,16}},crash_report,[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.1970.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,riak_ensemble_clock},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[kernel_sup,<0.1946.0>]},{messages,[]},{links,[<0.1947.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,124}],[]]} > {error_logger,{{2018,5,25},{11,30,16}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,riak_ensemble_clock}},{offender,[{pid,undefined},{name,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} > {error_logger,{{2018,5,25},{11,30,17}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.1945.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,riak_ensemble_clock}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.1944.0>]},{messages,[{'EXIT',<0.1946.0>,normal}]},{links,[<0.1944.0>,<0.1943.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,163}],[]]} > > I have two theories why it so, either not all deps in the release or don't start needed apps. How to understand the issue? I've read 'crash.dump', but it has a few info to get the reason. > > Thank you! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From gfborn@REDACTED Fri May 25 15:26:36 2018 From: gfborn@REDACTED (Grigory Fateyev) Date: Fri, 25 May 2018 16:26:36 +0300 Subject: [erlang-questions] From rebar2 to rebar3 In-Reply-To: <2CB89A5C-4002-454F-ABCA-45D363E9F7B0@uninet.ee> References: <2CB89A5C-4002-454F-ABCA-45D363E9F7B0@uninet.ee> Message-ID: Taavi, thank you! Your advice is very close to the reason. At least I got another error related to eleveldb. 2018-05-25 15:42 GMT+03:00 : > This log indicates, that > > failure is in ?on_load_function_fail?, which tries to load > ?riak_ensemble_clock' > - this is module trying to load NIF i.e. natively implemented function > i.e. C-compiled module > - quite likely something bad had happened and C module is not compiled > correctly and is not present > > Google indicates, that people have struggled with similar problems before > and had to help rebar configuration > > https://github.com/basho/riak_ensemble/issues/122 > > Maybe this will help you further.. > > best regards, > taavi > > > > On 25 May 2018, at 15:22, Grigory Fateyev wrote: > > > > Hello! > > > > We have a big project with a lot of dependencies and legacy code. I have > a task to move it to rebar3. I could compile and make a release, but > couldn't start it. It returns me: > > > > {error_logger,{{2018,5,25},{11,30,16}},crash_report,[[{ > initial_call,{supervisor,kernel,['Argument__1']}},{pid, > <0.1970.0>},{registered_name,[]},{error_info,{exit,{on_load_ > function_failed,riak_ensemble_clock},[{gen_server,init_it,6, > [{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_ > do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ > ancestors,[kernel_sup,<0.1946.0>]},{messages,[]},{links,[<0. > 1947.0>]},{dictionary,[]},{trap_exit,true},{status, > running},{heap_size,376},{stack_size,27},{reductions,124}],[]]} > > {error_logger,{{2018,5,25},{11,30,16}},supervisor_report,[ > {supervisor,{local,kernel_sup}},{errorContext,start_error},{ > reason,{on_load_function_failed,riak_ensemble_clock}},{ > offender,[{pid,undefined},{name,kernel_safe_sup},{mfargs, > {supervisor,start_link,[{local,kernel_safe_sup},kernel, > safe]}},{restart_type,permanent},{shutdown,infinity} > ,{child_type,supervisor}]}]} > > {error_logger,{{2018,5,25},{11,30,17}},crash_report,[[{ > initial_call,{application_master,init,['Argument__1',' > Argument__2','Argument__3','Argument__4']}},{pid,<0.1945. > 0>},{registered_name,[]},{error_info,{exit,{{shutdown,{ > failed_to_start_child,kernel_safe_sup,{on_load_function_ > failed,riak_ensemble_clock}}},{kernel,start,[normal,[]]}},[{ > application_master,init,4,[{file,"application_master.erl"} > ,{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib. > erl"},{line,239}]}]}},{ancestors,[<0.1944.0>]},{ > messages,[{'EXIT',<0.1946.0>,normal}]},{links,[<0.1944.0>,< > 0.1943.0>]},{dictionary,[]},{trap_exit,true},{status, > running},{heap_size,376},{stack_size,27},{reductions,163}],[]]} > > > > I have two theories why it so, either not all deps in the release or > don't start needed apps. How to understand the issue? I've read > 'crash.dump', but it has a few info to get the reason. > > > > Thank you! > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From heinz@REDACTED Fri May 25 15:38:56 2018 From: heinz@REDACTED (Heinz N. Gies) Date: Fri, 25 May 2018 15:38:56 +0200 Subject: [erlang-questions] From rebar2 to rebar3 In-Reply-To: References: <2CB89A5C-4002-454F-ABCA-45D363E9F7B0@uninet.ee> Message-ID: Hi Gregory, You can use the riak_* components from the riak_core_ng/Kyorai fork [1] they all build with both modern erlang and rebar3 also exist as hex packages (if you are into that kind of thing). Cheers, Heinz [1] http://github.com/Kyorai/ > On 25. May 2018, at 15:26, Grigory Fateyev wrote: > > Taavi, thank you! > > Your advice is very close to the reason. At least I got another error related to eleveldb. > > 2018-05-25 15:42 GMT+03:00 >: > This log indicates, that > > failure is in ?on_load_function_fail?, which tries to load ?riak_ensemble_clock' > - this is module trying to load NIF i.e. natively implemented function i.e. C-compiled module > - quite likely something bad had happened and C module is not compiled correctly and is not present > > Google indicates, that people have struggled with similar problems before > and had to help rebar configuration > > https://github.com/basho/riak_ensemble/issues/122 > > Maybe this will help you further.. > > best regards, > taavi > > > > On 25 May 2018, at 15:22, Grigory Fateyev > wrote: > > > > Hello! > > > > We have a big project with a lot of dependencies and legacy code. I have a task to move it to rebar3. I could compile and make a release, but couldn't start it. It returns me: > > > > {error_logger,{{2018,5,25},{11,30,16}},crash_report,[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.1970.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,riak_ensemble_clock},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[kernel_sup,<0.1946.0>]},{messages,[]},{links,[<0.1947.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,124}],[]]} > > {error_logger,{{2018,5,25},{11,30,16}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,riak_ensemble_clock}},{offender,[{pid,undefined},{name,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} > > {error_logger,{{2018,5,25},{11,30,17}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.1945.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,riak_ensemble_clock}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.1944.0>]},{messages,[{'EXIT',<0.1946.0>,normal}]},{links,[<0.1944.0>,<0.1943.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,163}],[]]} > > > > I have two theories why it so, either not all deps in the release or don't start needed apps. How to understand the issue? I've read 'crash.dump', but it has a few info to get the reason. > > > > Thank you! > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From essen@REDACTED Fri May 25 20:24:38 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Fri, 25 May 2018 20:24:38 +0200 Subject: [erlang-questions] Processes get stuck in prim_inet:recv0 Message-ID: <242bb42d-7e88-c760-d31c-fd13a82f665a@ninenines.eu> Hello, I have a container that's most likely subtly broken (most recent Arch Linux on a slightly older Arch Linux) and I'm not sure why. It only started happening after I rebuilt it a few days ago, the difference between before/after is more recent packages on the container and little else. It has a symptom that's consistent across all Erlang versions I'm testing (most recent patch releases of 19.*, 20.*, 21.0-rc1 and 20.3.6 compiled in native mode, and all are compiled on the same system with kerl): the process gets stuck in prim_inet:recv0 and it makes CT fail with a timetrap timeout[1]. It happens a few times per test runs, at a rate of maybe 2 or 3 times per hour. Have you seen this before? If yes how do I fix this? Is there any useful data I can get that would help? Thanks, [1] https://builds.ninenines.eu/logs/cowboy-prs/79/archlinux/ct_run.ct_cowboy@REDACTED/ninenines.cowboy-prs.rfc7230_SUITE.logs/run.2018-05-24_15.33.18/rfc7230_suite.timeout_after_request_line.2283.html -- Lo?c Hoguin https://ninenines.eu From max.lapshin@REDACTED Sat May 26 03:38:03 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 26 May 2018 04:38:03 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: Ok, will check with reducing buffer. We have put 2 MB and even 16 MB because without it, we got packet drops -------------- next part -------------- An HTML attachment was scrubbed... URL: From dane@REDACTED Sat May 26 04:35:34 2018 From: dane@REDACTED (Maxim Fedorov) Date: Sat, 26 May 2018 02:35:34 +0000 Subject: [erlang-questions] High Priority processes not handling system tasks Message-ID: <8CCE3D2E-CAD0-4A45-A7C8-6A4779104564@contoso.com> G?Day, Below is a chunk of code that does not look bad at a first glance, but? start() -> Pid = spawn(?MODULE, high, []), io:format("Starting... "), erlang:check_process_code(Pid, rec), io:format("collected~n"). high() -> erlang:process_flag(priority, high), endless(10). endless(X) -> endless(X+1). start() never returns, because check_process_code schedules a normal priority system task against high priority process. It seems to be logical, but leads to some really weird effects. For example, code:soft_purge() hangs (in erts_code_purger, cpc_receive), thus whole code_server gen_server hangs, and this paralyses almost everything (a separate issue - cpc_receive has no receive timeout at all, possibly stalling whole VM). Is it designed behaviour? I have quite a straightforward solution (patch for erl_process.c), but I might now understand consequences. Applies to R19+ (with 'new purge strategy' enabled). Older releases (eg. R16B) are not affected. Regards, Maxim Fedorov From lukas@REDACTED Sat May 26 08:17:52 2018 From: lukas@REDACTED (Lukas Larsson) Date: Sat, 26 May 2018 08:17:52 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: On Sat, May 26, 2018 at 3:38 AM, Max Lapshin wrote: > Ok, will check with reducing buffer. > > We have put 2 MB and even 16 MB because without it, we got packet drops > You could also try raising the sbct limit to over 2 MB. i.e. %% Raise sbct limit +MBsbct 4096 +MBlmbcs 20480 or %% lower user-space buffer Common = [binary,{reuseaddr,true},{buffer,256*1024},{recbuf,2*1024*1024}, inet,{read_packets,100},{active,500}], > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickard@REDACTED Sat May 26 19:59:07 2018 From: rickard@REDACTED (Rickard Green) Date: Sat, 26 May 2018 19:59:07 +0200 Subject: [erlang-questions] High Priority processes not handling system tasks Message-ID: I don't know why this mail only ended up in the archive, but not in my mailbox... > G?Day, > Below is a chunk of code that does not look bad at a first > glance, but? > > start() -> > Pid = spawn(?MODULE, high, []), > io:format("Starting... "), > erlang:check_process_code(Pid, rec), > io:format("collected~n"). > > high() -> > erlang:process_flag(priority, high), > endless(10). > > endless(X) -> > endless(X+1). > > > start() never returns, because check_process_code schedules a normal priority > system task against high priority process. It seems to be logical, but leads to > some really weird effects. For example, code:soft_purge() hangs (in > erts_code_purger, cpc_receive), thus whole code_server gen_server hangs, > and this paralyses almost everything (a separate issue - cpc_receive has no > receive timeout at all, possibly stalling whole VM). > > Is it designed behaviour? I have quite a straightforward solution (patch for > erl_process.c), but I might now understand consequences. > > Applies to R19+ (with 'new purge strategy' enabled). Older releases (eg. R16B) > are not affected. > > > Regards, > > Maxim Fedorov Yes this is how the process priority scheme was intentionally designed (for better or for worse). 'high' and 'max' process priority work are alway selected before lower process priority work. Changing the behaviour of system tasks so that lower priority tasks are interleaved with 'high' and/or 'max' priority work essentially introduce intentional priority inversion. Utilization of 'high' and 'max' priority has to be done with outmost care. Using high or max priorities on processes that hogs the cpu very likely will get you into trouble. The documentation of process_flag(priority, _) is also quite clear on this. This question comes up every now and then. In the past we have always rejected proposals for priory tweeks like this. I'm not claiming this priority scheme is the ultimate scheme, but this is what we have and I think we should stick with it as it is or redesign it from scratch. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From by@REDACTED Sun May 27 03:03:15 2018 From: by@REDACTED (by) Date: Sun, 27 May 2018 09:03:15 +0800 Subject: [erlang-questions] OTP Source File md5 checksum incorrect Message-ID: Hi, I am trying to download the OTP Source File from erlang.org under: http://www.erlang.org/downloads And After the download, I do md5 calculate for the file, and find the result not equal to which erlang.org provides. And I found that the md5 checksum which erlang.org provides is for tar.gz file, but what I downloaded is a tar file. Is there anyway I can download the tar.gz file and do the compare or maybe I can get the md5 checksum of the tar file? Thanks, Yao -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun May 27 09:20:33 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Sun, 27 May 2018 09:20:33 +0200 Subject: [erlang-questions] OTP Source File md5 checksum incorrect In-Reply-To: References: Message-ID: <07d8d014-0478-7a8a-322a-b42da16ad3d1@ninenines.eu> You did not download a tar file, you downloaded a tar.gz and your browser/client automatically gunzipped it. It's due to the HTTP headers in the response, it's sent back as an application/x-tar with content-encoding x-gzip, and some browsers save that decoded. You can use a different client to get the actual tar.gz, for example curl. To avoid the issue erlang.org should not send the content-encoding header in the response. On 05/27/2018 03:03 AM, by wrote: > Hi, > > I am trying to download the OTP Source File from erlang.org > under: > http://www.erlang.org/downloads > > And After the download, I do md5 calculate for the file, and find the > result not equal to which erlang.org ?provides. > > And I found that the md5 checksum which erlang.org > provides is for tar.gz file, but what I downloaded is a tar file. > > Is there anyway I can download the tar.gz file and do the compare or > maybe I can get the md5 checksum of the tar file? > > Thanks, > Yao > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From by@REDACTED Sun May 27 13:51:12 2018 From: by@REDACTED (by) Date: Sun, 27 May 2018 19:51:12 +0800 Subject: [erlang-questions] OTP Source File md5 checksum incorrect In-Reply-To: <07d8d014-0478-7a8a-322a-b42da16ad3d1@ninenines.eu> References: <07d8d014-0478-7a8a-322a-b42da16ad3d1@ninenines.eu> Message-ID: Yes, you are right. I get the gzipped file with curl. Thanks Yao > ? 2018?5?27??15:20?Lo?c Hoguin ??? > > You did not download a tar file, you downloaded a tar.gz and your browser/client automatically gunzipped it. > > It's due to the HTTP headers in the response, it's sent back as an application/x-tar with content-encoding x-gzip, and some browsers save that decoded. You can use a different client to get the actual tar.gz, for example curl. > > To avoid the issue erlang.org should not send the content-encoding header in the response. > >> On 05/27/2018 03:03 AM, by wrote: >> Hi, >> I am trying to download the OTP Source File from erlang.org under: >> http://www.erlang.org/downloads >> And After the download, I do md5 calculate for the file, and find the result not equal to which erlang.org provides. >> And I found that the md5 checksum which erlang.org provides is for tar.gz file, but what I downloaded is a tar file. >> Is there anyway I can download the tar.gz file and do the compare or maybe I can get the md5 checksum of the tar file? >> Thanks, >> Yao >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- > Lo?c Hoguin > https://ninenines.eu From bchesneau@REDACTED Sun May 27 16:24:32 2018 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sun, 27 May 2018 16:24:32 +0200 Subject: [erlang-questions] webrtc lib In-Reply-To: References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: understood :) are you using the latest version of erlang for dtls or still the fork in flussonic btw? On Sun, May 20, 2018 at 9:38 PM, Max Lapshin wrote: > There is an obvious problem: right now webrtc implementation is deeply > merged with flussonic internals like: > > websocket_info({dtls, key_material, #srtp_params{} = SrtpParams}, Req, > #publish{stream_name = StreamName} = State) -> > % events:debug("Got SRTP params: ~p", [SrtpParams]), > % {ok, DTS} = live_stream:getattr(StreamName, last_dts), > % Now = erlang:system_time(milli_seconds), > MI = live_stream:media_info(StreamName), > > > live_stream is a internal module and not supposed to be exported to > public-available library. > > It will take some time to extract. > That's totally understdanable -- Sent from my Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sun May 27 18:14:20 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 27 May 2018 19:14:20 +0300 Subject: [erlang-questions] webrtc lib In-Reply-To: References: <56B505B7-ABC6-4361-8658-92040281C694@gmail.com> Message-ID: Still fork. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gordeev.vladimir.v@REDACTED Mon May 28 08:35:27 2018 From: gordeev.vladimir.v@REDACTED (Vladimir Gordeev) Date: Mon, 28 May 2018 09:35:27 +0300 Subject: [erlang-questions] [ANN] Batiscaph seeing tool v0.1.1: visualization of the processes on the map Message-ID: Hello everyone! I was working on seeing tool for Erlang for some time. The purpose of this tool is to visualize what's happening on your Erlang node, help to understand your code. Observer included in erlang distribution shows you current state of the system. The tool I'm working on persists changes and show how things changed over time. The idea is to build a map where you can navigate around your processes and ports, just like you do it in Google Maps. Visualize trace events, logs and other information. It's still in the early stage and not that useful for production yet. I thought that I should share what I've already got anyway. I've recorded demo video explaining how it's used: https://www.youtube.com/watch?v=VNr7o9eg4Ck Github repo: https://github.com/vladimir-vg/batiscaph There you can find an instruction how to play with it locally. Would love to hear your feedback, thoughts, comments. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ?????? ?????? ?? 2018-05-26 11-35-15.png Type: image/png Size: 41742 bytes Desc: not available URL: From stamarit@REDACTED Mon May 28 12:03:22 2018 From: stamarit@REDACTED (Salvador Tamarit) Date: Mon, 28 May 2018 12:03:22 +0200 Subject: [erlang-questions] WFLP 2018: Second Call for Papers Message-ID: ========================================================= WFLP 2018: Second Call for Papers ========================================================= 26th International Workshop on Functional and (constraint) Logic Programming WFLP 2018 http://ppdp-lopstr-18.cs.uni-frankfurt.de/wflp18.html Frankfurt, Germany, September 6, 2018 (co-located with PPDP 2018 and LOPSTR 2018) Important Dates Abstract submission: June 27, 2018 Paper submission: July 2, 2018 Notification: July 27, 2018 Camera-ready: August 24, 2018 Workshop: September 6, 2018 WFLP 2018 The international Workshop on Functional and (constraint) Logic Programming (WFLP) aims at bringing together researchers, students, and practitioners interested in functional programming, logic programming, and their integration. WFLP has a reputation for being a lively and friendly forum, and it is open for presenting and discussing work in progress, technical contributions, experience reports, experiments, reviews, and system descriptions. The 26th International Workshop on Functional and (constraint) Logic Programming (WFLP 2018) will be held at the Goethe-University Frankfurt am Main, Germany. Previous WFLP editions were WFLP 2017 (W?rzburg, Germany), WFLP 2016 (Leipzig, Germany), WFLP 2014 (Wittenberg, Germany), WFLP 2013 (Kiel, Germany), WFLP 2012 (Nagoya, Japan), WFLP 2011 (Odense, Denmark), WFLP 2010 (Madrid, Spain), WFLP 2009 (Brasilia, Brazil), WFLP 2008 (Siena, Italy), WFLP 2007 (Paris, France), WFLP 2006 (Madrid, Spain), WCFLP 2005 (Tallinn, Estonia), WFLP 2004 (Aachen, Germany), WFLP 2003 (Valencia, Spain), WFLP 2002 (Grado, Italy), WFLP 2001 (Kiel, Germany), WFLP 2000 (Benicassim, Spain), WFLP'99 (Grenoble, France), WFLP'98 (Bad Honnef, Germany), WFLP'97 (Schwarzenberg, Germany), WFLP'96 (Marburg, Germany), WFLP'95 (Schwarzenberg, Germany), WFLP'94 (Schwarzenberg, Germany), WFLP'93 (Rattenberg, Germany), and WFLP'92 (Karlsruhe, Germany). WFLP 2018 will be co-located with PPDP 2018 (International Symposium on Principles and Practice of Declarative Programming) and LOPSTR 2018 (International Symposium on Logic-Based Program Synthesis and Transformation. Topics The topics of interest cover all aspects of functional and logic programming. They include (but are not limited to): * Functional programming * Logic programming * Constraint programming * Deductive databases, data mining * Extensions of declarative languages, objects * Multi-paradigm declarative programming * Foundations, semantics, non-monotonic reasoning, dynamics * Parallelism, concurrency * Program analysis, abstract interpretation * program and model manipulation * Program transformation, partial evaluation, meta-programming * Specification, verification * Debugging, testing * Knowledge representation, machine learning * Interaction of declarative programming with other formalisms * Implementation of declarative languages * Advanced programming environments and tools * Software techniques for declarative programming * Applications The primary focus is on new and original research results, but submissions describing innovative products, prototypes under development, application systems, or interesting experiments (e.g., benchmarks) are also encouraged. Survey papers that present some aspects of the above topics from a new perspective, and experience reports are also welcome. Papers must be written and presented in English. Work that already appeared in unpublished or informally published workshop proceedings may be submitted (please contact the PC chair in case of questions). Submission Guidelines Submission is via Easychair submission website for WFLP 2018: https://easychair.org/conferences/?conf=wflp2018 Authors are invited to submit papers in the following categories: + Regular research paper + Work-in-progress report + System description Regular research papers must describe original work, be written and presented in English, and must not substantially overlap with papers that have been formally published or that are simultaneously submitted to a journal, conference, or workshop with formal proceedings. They will be judged on the basis of significance, relevance, correctness, originality, and clarity. For work-in-progress reports and system descriptions, less formal rules apply, and presentation-only submissions (talk and discussion, but no paper in the formal proceedings) are possible. Please contact the PC chair with any questions. All submissions must be formatted in the Lecture Notes in Computer Science style. Submissions cannot exceed 15 pages including references but excluding well-marked appendices not intended for publication. Reviewers are not required to read the appendices, and thus papers should be intelligible without them. However, all submissions (especially work-in-progress reports and system descriptions) may be considerably shorter than 15 pages. Proceedings All papers accepted for presentation at the conference will be published in informal proceedings publicly available at the Computing Research Repository. According to the program committee reviews, submissions can be directly accepted for publication in the formal post-conference proceedings. The formal post-conference proceedings will be published in both electronic and paper formats by Springer in the Lecture Notes in Computer Science series. After the conference, all authors accepted only for presentation will be invited to revise and/or extend their submissions in the light of the feedback solicited at the conference. Then, after another round of reviewing, these revised papers may also be published in the formal proceedings. Therefore, all accepted papers will be published in open-access, and the authors can also decide to publish their work in the Springer LNCS formal proceedings. Program Committee + Slim Abdennadher, German University in Cairo, Egypt + Maria Alpuente, Universitat Polit?cnica de Val?ncia, Spain + Sergio Antoy, Portland State University, USA + Olaf Chitil, University of Kent, UK + Maria del Mar Gallardo, Universidad de M?laga, Spain + Michael Hanus, University of Kiel, Germany + Herbert Kuchen, University of M?nster, Germany + Kostis Sagonas, Uppsala University, Sweden + Tom Schrijvers, Katholieke Universiteit Leuven, Belgium + Sibylle Schwarz, HTWK Leipzig, Germany + Martina Seidl, Johannes Kepler University Linz, Austria + Dietmar Seipel, University of W?rzburg, Germany + Josep Silva, Universitat Polit?cnica de Val?ncia, Spain (Program Chair) + Salvador Tamarit, Universidad Polit?cnica de Madrid, Spain + Janis Voigtl?nder, University of Duisburg-Essen, Germany + Johannes Waldmann, HTWK Leipzig, Germany Organizing Committee David Sabel (General Chair), Computer Science Institute Goethe-University Frankfurt am Main, Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.merle@REDACTED Mon May 28 23:42:29 2018 From: s.merle@REDACTED (=?UTF-8?Q?S=C3=A9bastien_Merle?=) Date: Mon, 28 May 2018 23:42:29 +0200 Subject: [erlang-questions] Logger error with OTP 21.0-rc1 Message-ID: Hi, I am porting the grisp software (grisp.org) to OTP 21.0-rc1. Everything build and deploy just fine but when running Erlang on the board the console is swamped with error like: `(no logger present) unexpected logger message: {log,error,"Error in process ~p with exit value:~n~p~n",[<0.39.0>,{badarg,[{ets,lookup,[logger,proc_lib],[]},{logger_config,allow,3,[{file,"logger_config.erl"},{...` I saw someone else complaining about this same error on the thread about 21.0-rc1 but the only response was to use a clean build, and I already built OTP from a fresh checkout. Any idea what could be the cause ? Maybe this is a know issue or the new logger API requires some configuration or service to be started explicitly ? Thank you very much. Sebastien Merle, Peer Stritzinger GmbH. -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Mon May 28 23:46:56 2018 From: igor.clark@REDACTED (Igor Clark) Date: Mon, 28 May 2018 22:46:56 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF Message-ID: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Hi folks, hope all well, I have a NIF which very occasionally segfaults, intermittently and apparently unpredictably, bringing down the VM. I've spent a bunch of time tracing allocation and dereferencing problems in my NIF code, and I've got rid of what seems like 99%+ of the problems - but it still occasionally happens, and I'm having trouble tracing further, because the crash logs show the crashed threads as doing things like these: (each one taken from a separate log where it's the only crashed thread) > Thread 40 Crashed:: 8_scheduler > 0?? beam.smp????????????????????? ??? 0x000000001c19980b process_main > + 1570 > > Thread 5 Crashed:: 3_scheduler > 0?? beam.smp????????????????????? ??? 0x000000001c01d80b process_main > + 1570 > > Thread 7 Crashed:: 5_scheduler > 0?? beam.smp????????????????????? ??? 0x000000001baff0b8 > lists_member_2 + 63 > > Thread 3 Crashed:: 1_scheduler > 0?? beam.smp????????????????????? ??? 0x000000001d4b780b process_main > + 1570 > > Thread 5 Crashed:: 3_scheduler > 0?? beam.smp????????????????????? ??? 0x000000001fcf280b process_main > + 1570 > > Thread 6 Crashed:: 4_scheduler > 0?? beam.smp????????????????????? ??? 0x000000001ae290b8 > lists_member_2 + 63 I'm very confident that the problems are in my code, not in the scheduler ;-) But without more detail, I don't know how to trace where they're happening. When they do, there are sometimes other threads doing things in my code (maybe 20% of the time) - but mostly not, and on the occasions when they are, I've been unable to see what the problem might be on the lines referenced. It seems like it's some kind of cross-thread data access issue, but I don't know how to track it down. Some more context about what's going on. My NIF load() function starts a thread which passes a callback function to a library that talks to some hardware, which calls the callback when it has a message. It's a separate thread because the library only calls back to the thread that initialized it; when I ran it directly in NIF load(), it didn't call back, but in the VM-managed thread, it works as expected. The thread sits and waits for stuff to happen, and callbacks come when they should. I use enif_thread_create/enif_thread_opts_create to start the thread, and use enif_alloc/enif_free everywhere. I keep a static pointer in the NIF to a couple of members of the state struct, as that seems the only way to reference them in the callback function. The struct is kept in NIF private data: I pass **priv from load() to the thread_main function, allocate the state struct using enif_alloc in thread_main, and set priv pointing to the state struct, also in the thread. Other NIF functions do access members of the state struct, but only ever through enif_priv_data( env ). The vast majority of the time it all works perfectly, humming along very nicely, but every now and then, without any real pattern I can see, it just segfaults and the VM comes down. It's only happened 3 times in the last 20+ hours of working on the app, testing & running all the while, doing VM starts, stops, code reloads, etc. But when it happens, it's kind of a showstopper, and I'd really like to nail it down. This is all happening in Erlang 20.3.4 on MacOS 10.12.6 / Apple LLVM version 9.0.0 (clang-900.0.38). Any ideas on how/where to look next to try to track this down? Hope it's not something structural in the above which just won't work. Cheers, Igor From sergej.jurecko@REDACTED Tue May 29 05:16:52 2018 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Tue, 29 May 2018 05:16:52 +0200 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: On macOS there is a quick way to get a stack trace if you compiled with debug symbols. Open /Applications/Utilities/Console Go to: User Reports You will see beam.smp in there if it crashed. Click on it and you get a report what every thread was calling at the time of crash. Regards, Sergej > On 28 May 2018, at 23:46, Igor Clark wrote: > > Hi folks, hope all well, > > I have a NIF which very occasionally segfaults, intermittently and apparently unpredictably, bringing down the VM. I've spent a bunch of time tracing allocation and dereferencing problems in my NIF code, and I've got rid of what seems like 99%+ of the problems - but it still occasionally happens, and I'm having trouble tracing further, because the crash logs show the crashed threads as doing things like these: (each one taken from a separate log where it's the only crashed thread) > > >> Thread 40 Crashed:: 8_scheduler >> 0 beam.smp 0x000000001c19980b process_main + 1570 >> >> Thread 5 Crashed:: 3_scheduler >> 0 beam.smp 0x000000001c01d80b process_main + 1570 >> >> Thread 7 Crashed:: 5_scheduler >> 0 beam.smp 0x000000001baff0b8 lists_member_2 + 63 >> >> Thread 3 Crashed:: 1_scheduler >> 0 beam.smp 0x000000001d4b780b process_main + 1570 >> >> Thread 5 Crashed:: 3_scheduler >> 0 beam.smp 0x000000001fcf280b process_main + 1570 >> >> Thread 6 Crashed:: 4_scheduler >> 0 beam.smp 0x000000001ae290b8 lists_member_2 + 63 > > > I'm very confident that the problems are in my code, not in the scheduler ;-) But without more detail, I don't know how to trace where they're happening. When they do, there are sometimes other threads doing things in my code (maybe 20% of the time) - but mostly not, and on the occasions when they are, I've been unable to see what the problem might be on the lines referenced. > > It seems like it's some kind of cross-thread data access issue, but I don't know how to track it down. > > Some more context about what's going on. My NIF load() function starts a thread which passes a callback function to a library that talks to some hardware, which calls the callback when it has a message. It's a separate thread because the library only calls back to the thread that initialized it; when I ran it directly in NIF load(), it didn't call back, but in the VM-managed thread, it works as expected. The thread sits and waits for stuff to happen, and callbacks come when they should. > > I use enif_thread_create/enif_thread_opts_create to start the thread, and use enif_alloc/enif_free everywhere. I keep a static pointer in the NIF to a couple of members of the state struct, as that seems the only way to reference them in the callback function. The struct is kept in NIF private data: I pass **priv from load() to the thread_main function, allocate the state struct using enif_alloc in thread_main, and set priv pointing to the state struct, also in the thread. Other NIF functions do access members of the state struct, but only ever through enif_priv_data( env ). > > The vast majority of the time it all works perfectly, humming along very nicely, but every now and then, without any real pattern I can see, it just segfaults and the VM comes down. It's only happened 3 times in the last 20+ hours of working on the app, testing & running all the while, doing VM starts, stops, code reloads, etc. But when it happens, it's kind of a showstopper, and I'd really like to nail it down. > > This is all happening in Erlang 20.3.4 on MacOS 10.12.6 / Apple LLVM version 9.0.0 (clang-900.0.38). > > Any ideas on how/where to look next to try to track this down? Hope it's not something structural in the above which just won't work. > > Cheers, > Igor > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From igor.clark@REDACTED Tue May 29 11:04:22 2018 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 29 May 2018 10:04:22 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: Thanks Sergej - that's where I got the thread reports I pasted in below, from e.g. 'beam.smp_2018-05-28-212735_Igor-Clarks-iMac.crash'. Each log says the only crashed thread was a scheduler thread, for example "8_scheduler" running "process_main" in the case of the first one below. This is how I tracked down a bunch of errors in my own code, but the only ones that still happen are in the scheduler, according to the Console crash logs. The thing is, it seems really unlikely that a VM running my NIF code would just happen to be crashing in the scheduler rather than my code(!) - so that's what I'm trying to work out, how to find out what's actually going on, given that the log tells me the crashed thread is running "process_main" or 'lists_member_2'. Any suggestions welcome! Cheers, Igor On 29/05/2018 04:16, Sergej Jure?ko wrote: > On macOS there is a quick way to get a stack trace if you compiled with debug symbols. > Open /Applications/Utilities/Console > Go to: User Reports > > You will see beam.smp in there if it crashed. Click on it and you get a report what every thread was calling at the time of crash. > > > Regards, > Sergej > >> On 28 May 2018, at 23:46, Igor Clark wrote: >> >> Hi folks, hope all well, >> >> I have a NIF which very occasionally segfaults, intermittently and apparently unpredictably, bringing down the VM. I've spent a bunch of time tracing allocation and dereferencing problems in my NIF code, and I've got rid of what seems like 99%+ of the problems - but it still occasionally happens, and I'm having trouble tracing further, because the crash logs show the crashed threads as doing things like these: (each one taken from a separate log where it's the only crashed thread) >> >> >>> Thread 40 Crashed:: 8_scheduler >>> 0 beam.smp 0x000000001c19980b process_main + 1570 >>> >>> Thread 5 Crashed:: 3_scheduler >>> 0 beam.smp 0x000000001c01d80b process_main + 1570 >>> >>> Thread 7 Crashed:: 5_scheduler >>> 0 beam.smp 0x000000001baff0b8 lists_member_2 + 63 >>> >>> Thread 3 Crashed:: 1_scheduler >>> 0 beam.smp 0x000000001d4b780b process_main + 1570 >>> >>> Thread 5 Crashed:: 3_scheduler >>> 0 beam.smp 0x000000001fcf280b process_main + 1570 >>> >>> Thread 6 Crashed:: 4_scheduler >>> 0 beam.smp 0x000000001ae290b8 lists_member_2 + 63 >> >> I'm very confident that the problems are in my code, not in the scheduler ;-) But without more detail, I don't know how to trace where they're happening. When they do, there are sometimes other threads doing things in my code (maybe 20% of the time) - but mostly not, and on the occasions when they are, I've been unable to see what the problem might be on the lines referenced. >> >> It seems like it's some kind of cross-thread data access issue, but I don't know how to track it down. >> >> Some more context about what's going on. My NIF load() function starts a thread which passes a callback function to a library that talks to some hardware, which calls the callback when it has a message. It's a separate thread because the library only calls back to the thread that initialized it; when I ran it directly in NIF load(), it didn't call back, but in the VM-managed thread, it works as expected. The thread sits and waits for stuff to happen, and callbacks come when they should. >> >> I use enif_thread_create/enif_thread_opts_create to start the thread, and use enif_alloc/enif_free everywhere. I keep a static pointer in the NIF to a couple of members of the state struct, as that seems the only way to reference them in the callback function. The struct is kept in NIF private data: I pass **priv from load() to the thread_main function, allocate the state struct using enif_alloc in thread_main, and set priv pointing to the state struct, also in the thread. Other NIF functions do access members of the state struct, but only ever through enif_priv_data( env ). >> >> The vast majority of the time it all works perfectly, humming along very nicely, but every now and then, without any real pattern I can see, it just segfaults and the VM comes down. It's only happened 3 times in the last 20+ hours of working on the app, testing & running all the while, doing VM starts, stops, code reloads, etc. But when it happens, it's kind of a showstopper, and I'd really like to nail it down. >> >> This is all happening in Erlang 20.3.4 on MacOS 10.12.6 / Apple LLVM version 9.0.0 (clang-900.0.38). >> >> Any ideas on how/where to look next to try to track this down? Hope it's not something structural in the above which just won't work. >> >> Cheers, >> Igor >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From erlangsiri@REDACTED Tue May 29 11:21:15 2018 From: erlangsiri@REDACTED (Siri Hansen) Date: Tue, 29 May 2018 11:21:15 +0200 Subject: [erlang-questions] Logger error with OTP 21.0-rc1 In-Reply-To: References: Message-ID: Hi Sebastien, which script/boot file do you use? Is it built with the new OTP version? Kind Regards /siri@REDACTED 2018-05-28 23:42 GMT+02:00 S?bastien Merle : > Hi, > > I am porting the grisp software (grisp.org) to OTP 21.0-rc1. > Everything build and deploy just fine but when running Erlang on the board > the console is swamped with error like: > > `(no logger present) unexpected logger message: {log,error,"Error in > process ~p with exit value:~n~p~n",[<0.39.0>,{badarg,[{ets,lookup,[logger, > proc_lib],[]},{logger_config,allow,3,[{file,"logger_config.erl"},{...` > > I saw someone else complaining about this same error on the thread about > 21.0-rc1 but the only response was to use a clean build, and I already > built OTP from a fresh checkout. > > Any idea what could be the cause ? Maybe this is a know issue or the new > logger API requires some configuration or service to be started explicitly ? > > Thank you very much. > > Sebastien Merle, > Peer Stritzinger GmbH. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Tue May 29 12:30:16 2018 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 29 May 2018 12:30:16 +0200 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: Have you tried to run your code in a debug emulator? https://github.com/erlang/otp/blob/master/HOWTO/INSTALL.md#how-to-build-a-debug-enabled-erlang-runtime-system Since it seems to be segfaulting in lists:member/2, I would guess that your nif somehow builds an invalid list that later is used by lists:member/2. On Tue, May 29, 2018 at 11:04 AM, Igor Clark wrote: > Thanks Sergej - that's where I got the thread reports I pasted in below, > from e.g. 'beam.smp_2018-05-28-212735_Igor-Clarks-iMac.crash'. > > Each log says the only crashed thread was a scheduler thread, for example > "8_scheduler" running "process_main" in the case of the first one below. > This is how I tracked down a bunch of errors in my own code, but the only > ones that still happen are in the scheduler, according to the Console crash > logs. > > The thing is, it seems really unlikely that a VM running my NIF code would > just happen to be crashing in the scheduler rather than my code(!) - so > that's what I'm trying to work out, how to find out what's actually going > on, given that the log tells me the crashed thread is running > "process_main" or 'lists_member_2'. > > Any suggestions welcome! > > Cheers, > Igor > > > On 29/05/2018 04:16, Sergej Jure?ko wrote: > >> On macOS there is a quick way to get a stack trace if you compiled with >> debug symbols. >> Open /Applications/Utilities/Console >> Go to: User Reports >> >> You will see beam.smp in there if it crashed. Click on it and you get a >> report what every thread was calling at the time of crash. >> >> >> Regards, >> Sergej >> >> On 28 May 2018, at 23:46, Igor Clark wrote: >>> >>> Hi folks, hope all well, >>> >>> I have a NIF which very occasionally segfaults, intermittently and >>> apparently unpredictably, bringing down the VM. I've spent a bunch of time >>> tracing allocation and dereferencing problems in my NIF code, and I've got >>> rid of what seems like 99%+ of the problems - but it still occasionally >>> happens, and I'm having trouble tracing further, because the crash logs >>> show the crashed threads as doing things like these: (each one taken from a >>> separate log where it's the only crashed thread) >>> >>> >>> Thread 40 Crashed:: 8_scheduler >>>> 0 beam.smp 0x000000001c19980b process_main + >>>> 1570 >>>> >>>> Thread 5 Crashed:: 3_scheduler >>>> 0 beam.smp 0x000000001c01d80b process_main + >>>> 1570 >>>> >>>> Thread 7 Crashed:: 5_scheduler >>>> 0 beam.smp 0x000000001baff0b8 lists_member_2 >>>> + 63 >>>> >>>> Thread 3 Crashed:: 1_scheduler >>>> 0 beam.smp 0x000000001d4b780b process_main + >>>> 1570 >>>> >>>> Thread 5 Crashed:: 3_scheduler >>>> 0 beam.smp 0x000000001fcf280b process_main + >>>> 1570 >>>> >>>> Thread 6 Crashed:: 4_scheduler >>>> 0 beam.smp 0x000000001ae290b8 lists_member_2 >>>> + 63 >>>> >>> >>> I'm very confident that the problems are in my code, not in the >>> scheduler ;-) But without more detail, I don't know how to trace where >>> they're happening. When they do, there are sometimes other threads doing >>> things in my code (maybe 20% of the time) - but mostly not, and on the >>> occasions when they are, I've been unable to see what the problem might be >>> on the lines referenced. >>> >>> It seems like it's some kind of cross-thread data access issue, but I >>> don't know how to track it down. >>> >>> Some more context about what's going on. My NIF load() function starts a >>> thread which passes a callback function to a library that talks to some >>> hardware, which calls the callback when it has a message. It's a separate >>> thread because the library only calls back to the thread that initialized >>> it; when I ran it directly in NIF load(), it didn't call back, but in the >>> VM-managed thread, it works as expected. The thread sits and waits for >>> stuff to happen, and callbacks come when they should. >>> >>> I use enif_thread_create/enif_thread_opts_create to start the thread, >>> and use enif_alloc/enif_free everywhere. I keep a static pointer in the NIF >>> to a couple of members of the state struct, as that seems the only way to >>> reference them in the callback function. The struct is kept in NIF private >>> data: I pass **priv from load() to the thread_main function, allocate the >>> state struct using enif_alloc in thread_main, and set priv pointing to the >>> state struct, also in the thread. Other NIF functions do access members of >>> the state struct, but only ever through enif_priv_data( env ). >>> >>> The vast majority of the time it all works perfectly, humming along very >>> nicely, but every now and then, without any real pattern I can see, it just >>> segfaults and the VM comes down. It's only happened 3 times in the last 20+ >>> hours of working on the app, testing & running all the while, doing VM >>> starts, stops, code reloads, etc. But when it happens, it's kind of a >>> showstopper, and I'd really like to nail it down. >>> >>> This is all happening in Erlang 20.3.4 on MacOS 10.12.6 / Apple LLVM >>> version 9.0.0 (clang-900.0.38). >>> >>> Any ideas on how/where to look next to try to track this down? Hope it's >>> not something structural in the above which just won't work. >>> >>> Cheers, >>> Igor >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.merle@REDACTED Tue May 29 12:45:12 2018 From: s.merle@REDACTED (=?UTF-8?Q?S=C3=A9bastien_Merle?=) Date: Tue, 29 May 2018 12:45:12 +0200 Subject: [erlang-questions] Logger error with OTP 21.0-rc1 In-Reply-To: References: Message-ID: The release is built using rebar3: `{relx, [{release, {testproject, "0.1.0"}, [ testproject ]}]}.` and erlang is started on the board with: `erl.rtems -- -home . -pa . -root testproject -boot testproject/releases/0.1.0/testproject -kernel inetrc "./erl_inetrc"` Thank you for your help. Sebastien Merle, Peer Stritzinger GmbH. On Tue, May 29, 2018 at 11:21 AM, Siri Hansen wrote: > Hi Sebastien, > > which script/boot file do you use? Is it built with the new OTP version? > > Kind Regards > /siri@REDACTED > > 2018-05-28 23:42 GMT+02:00 S?bastien Merle : > >> Hi, >> >> I am porting the grisp software (grisp.org) to OTP 21.0-rc1. >> Everything build and deploy just fine but when running Erlang on the >> board the console is swamped with error like: >> >> `(no logger present) unexpected logger message: {log,error,"Error in >> process ~p with exit value:~n~p~n",[<0.39.0>,{badar >> g,[{ets,lookup,[logger,proc_lib],[]},{logger_config,allow, >> 3,[{file,"logger_config.erl"},{...` >> >> I saw someone else complaining about this same error on the thread about >> 21.0-rc1 but the only response was to use a clean build, and I already >> built OTP from a fresh checkout. >> >> Any idea what could be the cause ? Maybe this is a know issue or the new >> logger API requires some configuration or service to be started explicitly ? >> >> Thank you very much. >> >> Sebastien Merle, >> Peer Stritzinger GmbH. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Tue May 29 13:30:42 2018 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 29 May 2018 12:30:42 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: Thanks very much Lukas, I think the debug emulator could be what I'm looking for. The NIF only sometimes crashes on lists:member/2 - those log lines are all from different crashes (there's only one crashed thread each time), and sometimes it just crashes on process_main. So I think I might need the debug emulator to trace further. However I have a lot to learn about how to integrate C tooling with something so complex. When I run the debug emulator, does it just show more detailed info in stack traces, or will I need to attach gdb/lldb etc to find out what's going on? Is there any more info on how to set this all up? Also, not 100% sure how to run it, as I run my app with "rebar3 shell" from a release layout during development, or the same inside the NIF-specific app when trying to track problems down there. The doc you linked says: > To start the debug enabled runtime system execute: > > |$ $ERL_TOP/bin/cerl -debug| I realise these are more rebar3 than erlang questions, but I can't find much in the rebar3 docs about them: - How should I specify that rebar3 should run "cerl" instead of "erl" ? - Should I just add "-debug" in my "config/vm.args" or is there another way to do this? Thank you for your help! i On 29/05/2018 11:30, Lukas Larsson wrote: > Have you tried to run your code in a debug emulator? > https://github.com/erlang/otp/blob/master/HOWTO/INSTALL.md#how-to-build-a-debug-enabled-erlang-runtime-system > > > Since it seems to be segfaulting in lists:member/2, I would guess that > your nif somehow builds an invalid list that later is used by > lists:member/2. > > On Tue, May 29, 2018 at 11:04 AM, Igor Clark > wrote: > > Thanks Sergej - that's where I got the thread reports I pasted in > below, from e.g. 'beam.smp_2018-05-28-212735_Igor-Clarks-iMac.crash'. > > Each log says the only crashed thread was a scheduler thread, for > example "8_scheduler" running "process_main" in the case of the > first one below. This is how I tracked down a bunch of errors in > my own code, but the only ones that still happen are in the > scheduler, according to the Console crash logs. > > The thing is, it seems really unlikely that a VM running my NIF > code would just happen to be crashing in the scheduler rather than > my code(!) - so that's what I'm trying to work out, how to find > out what's actually going on, given that the log tells me the > crashed thread is running "process_main" or 'lists_member_2'. > > Any suggestions welcome! > > Cheers, > Igor > > > On 29/05/2018 04:16, Sergej Jure?ko wrote: > > On macOS there is a quick way to get a stack trace if you > compiled with debug symbols. > Open /Applications/Utilities/Console > Go to: User Reports > > You will see beam.smp in there if it crashed. Click on it and > you get a report what every thread was calling at the time of > crash. > > > Regards, > Sergej > > On 28 May 2018, at 23:46, Igor Clark > wrote: > > Hi folks, hope all well, > > I have a NIF which very occasionally segfaults, > intermittently and apparently unpredictably, bringing down > the VM. I've spent a bunch of time tracing allocation and > dereferencing problems in my NIF code, and I've got rid of > what seems like 99%+ of the problems - but it still > occasionally happens, and I'm having trouble tracing > further, because the crash logs show the crashed threads > as doing things like these: (each one taken from a > separate log where it's the only crashed thread) > > > Thread 40 Crashed:: 8_scheduler > 0? ?beam.smp 0x000000001c19980b process_main + 1570 > > Thread 5 Crashed:: 3_scheduler > 0? ?beam.smp 0x000000001c01d80b process_main + 1570 > > Thread 7 Crashed:: 5_scheduler > 0? ?beam.smp 0x000000001baff0b8 lists_member_2 + 63 > > Thread 3 Crashed:: 1_scheduler > 0? ?beam.smp 0x000000001d4b780b process_main + 1570 > > Thread 5 Crashed:: 3_scheduler > 0? ?beam.smp 0x000000001fcf280b process_main + 1570 > > Thread 6 Crashed:: 4_scheduler > 0? ?beam.smp 0x000000001ae290b8 lists_member_2 + 63 > > > I'm very confident that the problems are in my code, not > in the scheduler ;-) But without more detail, I don't know > how to trace where they're happening. When they do, there > are sometimes other threads doing things in my code (maybe > 20% of the time) - but mostly not, and on the occasions > when they are, I've been unable to see what the problem > might be on the lines referenced. > > It seems like it's some kind of cross-thread data access > issue, but I don't know how to track it down. > > Some more context about what's going on. My NIF load() > function starts a thread which passes a callback function > to a library that talks to some hardware, which calls the > callback when it has a message. It's a separate thread > because the library only calls back to the thread that > initialized it; when I ran it directly in NIF load(), it > didn't call back, but in the VM-managed thread, it works > as expected. The thread sits and waits for stuff to > happen, and callbacks come when they should. > > I use enif_thread_create/enif_thread_opts_create to start > the thread, and use enif_alloc/enif_free everywhere. I > keep a static pointer in the NIF to a couple of members of > the state struct, as that seems the only way to reference > them in the callback function. The struct is kept in NIF > private data: I pass **priv from load() to the thread_main > function, allocate the state struct using enif_alloc in > thread_main, and set priv pointing to the state struct, > also in the thread. Other NIF functions do access members > of the state struct, but only ever through enif_priv_data( > env ). > > The vast majority of the time it all works perfectly, > humming along very nicely, but every now and then, without > any real pattern I can see, it just segfaults and the VM > comes down. It's only happened 3 times in the last 20+ > hours of working on the app, testing & running all the > while, doing VM starts, stops, code reloads, etc. But when > it happens, it's kind of a showstopper, and I'd really > like to nail it down. > > This is all happening in Erlang 20.3.4 on MacOS 10.12.6 / > Apple LLVM version 9.0.0 (clang-900.0.38). > > Any ideas on how/where to look next to try to track this > down? Hope it's not something structural in the above > which just won't work. > > Cheers, > Igor > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.merle@REDACTED Tue May 29 14:04:47 2018 From: s.merle@REDACTED (=?UTF-8?Q?S=C3=A9bastien_Merle?=) Date: Tue, 29 May 2018 14:04:47 +0200 Subject: [erlang-questions] Logger error with OTP 21.0-rc1 In-Reply-To: References: Message-ID: After some unsuccessful investigation I re-read your message I realized I was still using 20.2 to release my application ! This was the issue , everything seems to work fin when using 21.0-rc1 for everything. Thank you very much. On Tue, May 29, 2018 at 12:45 PM, S?bastien Merle wrote: > The release is built using rebar3: > > `{relx, [{release, {testproject, "0.1.0"}, [ testproject ]}]}.` > > and erlang is started on the board with: > > `erl.rtems -- -home . -pa . -root testproject -boot > testproject/releases/0.1.0/testproject -kernel inetrc "./erl_inetrc"` > > Thank you for your help. > > Sebastien Merle, > Peer Stritzinger GmbH. > > > On Tue, May 29, 2018 at 11:21 AM, Siri Hansen > wrote: > >> Hi Sebastien, >> >> which script/boot file do you use? Is it built with the new OTP version? >> >> Kind Regards >> /siri@REDACTED >> >> 2018-05-28 23:42 GMT+02:00 S?bastien Merle : >> >>> Hi, >>> >>> I am porting the grisp software (grisp.org) to OTP 21.0-rc1. >>> Everything build and deploy just fine but when running Erlang on the >>> board the console is swamped with error like: >>> >>> `(no logger present) unexpected logger message: {log,error,"Error in >>> process ~p with exit value:~n~p~n",[<0.39.0>,{badar >>> g,[{ets,lookup,[logger,proc_lib],[]},{logger_config,allow,3, >>> [{file,"logger_config.erl"},{...` >>> >>> I saw someone else complaining about this same error on the thread about >>> 21.0-rc1 but the only response was to use a clean build, and I already >>> built OTP from a fresh checkout. >>> >>> Any idea what could be the cause ? Maybe this is a know issue or the new >>> logger API requires some configuration or service to be started explicitly ? >>> >>> Thank you very much. >>> >>> Sebastien Merle, >>> Peer Stritzinger GmbH. >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott_ribe@REDACTED Tue May 29 14:20:45 2018 From: scott_ribe@REDACTED (Scott Ribe) Date: Tue, 29 May 2018 06:20:45 -0600 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> What's most likely going on is that your NIF is corrupting memory that the scheduler uses. These kinds of corruptions are difficult to track down, because the crash can happen long after the bug. You should be running with all memory debugging aids turned on--this can get you a crash closer to the actual bug, if you're lucky maybe even at the bug. https://developer.apple.com/library/content/documentation/Performance/Conceptual/ManagingMemory/Articles/MallocDebug.html https://developer.apple.com/library/content/technotes/tn2124/_index.html -- Scott Ribe scott_ribe@REDACTED https://www.linkedin.com/in/scottribe/ > On May 29, 2018, at 3:04 AM, Igor Clark wrote: > > Thanks Sergej - that's where I got the thread reports I pasted in below, from e.g. 'beam.smp_2018-05-28-212735_Igor-Clarks-iMac.crash'. > > Each log says the only crashed thread was a scheduler thread, for example "8_scheduler" running "process_main" in the case of the first one below. This is how I tracked down a bunch of errors in my own code, but the only ones that still happen are in the scheduler, according to the Console crash logs. > > The thing is, it seems really unlikely that a VM running my NIF code would just happen to be crashing in the scheduler rather than my code(!) - so that's what I'm trying to work out, how to find out what's actually going on, given that the log tells me the crashed thread is running "process_main" or 'lists_member_2'. > > Any suggestions welcome! > > Cheers, > Igor > > On 29/05/2018 04:16, Sergej Jure?ko wrote: From lukas@REDACTED Tue May 29 14:45:46 2018 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 29 May 2018 14:45:46 +0200 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: I don't know how to make rebar3 run the debug emulator, but a quick and dirty trick that I do when all else fails is to copy the beam.debug.smp file over the beam.smp file. You probably also have to copy the erl_child_setup.debug file, that file should however have the .debug suffix remaining. So: cp bin/`erts/autoconf/config.guess`/beam.debug.smp path/to/release/erts-v.s.n/bin/beam.smp cp bin/`erts/autoconf/config.guess`/erl_child_setup.debug path/to/release/erts-v.s.n/bin/ On Tue, May 29, 2018 at 1:30 PM, Igor Clark wrote: > Thanks very much Lukas, I think the debug emulator could be what I'm > looking for. The NIF only sometimes crashes on lists:member/2 - those log > lines are all from different crashes (there's only one crashed thread each > time), and sometimes it just crashes on process_main. So I think I might > need the debug emulator to trace further. > > However I have a lot to learn about how to integrate C tooling with > something so complex. When I run the debug emulator, does it just show more > detailed info in stack traces, or will I need to attach gdb/lldb etc to > find out what's going on? Is there any more info on how to set this all up? > > Also, not 100% sure how to run it, as I run my app with "rebar3 shell" > from a release layout during development, or the same inside the > NIF-specific app when trying to track problems down there. The doc you > linked says: > > To start the debug enabled runtime system execute: > > $ $ERL_TOP/bin/cerl -debug > > > I realise these are more rebar3 than erlang questions, but I can't find > much in the rebar3 docs about them: > > - How should I specify that rebar3 should run "cerl" instead of "erl" ? > > - Should I just add "-debug" in my "config/vm.args" or is there another > way to do this? > > Thank you for your help! > i > > > On 29/05/2018 11:30, Lukas Larsson wrote: > > Have you tried to run your code in a debug emulator? https://github.com/ > erlang/otp/blob/master/HOWTO/INSTALL.md#how-to-build-a- > debug-enabled-erlang-runtime-system > > Since it seems to be segfaulting in lists:member/2, I would guess that > your nif somehow builds an invalid list that later is used by > lists:member/2. > > On Tue, May 29, 2018 at 11:04 AM, Igor Clark wrote: > >> Thanks Sergej - that's where I got the thread reports I pasted in below, >> from e.g. 'beam.smp_2018-05-28-212735_Igor-Clarks-iMac.crash'. >> >> Each log says the only crashed thread was a scheduler thread, for example >> "8_scheduler" running "process_main" in the case of the first one below. >> This is how I tracked down a bunch of errors in my own code, but the only >> ones that still happen are in the scheduler, according to the Console crash >> logs. >> >> The thing is, it seems really unlikely that a VM running my NIF code >> would just happen to be crashing in the scheduler rather than my code(!) - >> so that's what I'm trying to work out, how to find out what's actually >> going on, given that the log tells me the crashed thread is running >> "process_main" or 'lists_member_2'. >> >> Any suggestions welcome! >> >> Cheers, >> Igor >> >> >> On 29/05/2018 04:16, Sergej Jure?ko wrote: >> >>> On macOS there is a quick way to get a stack trace if you compiled with >>> debug symbols. >>> Open /Applications/Utilities/Console >>> Go to: User Reports >>> >>> You will see beam.smp in there if it crashed. Click on it and you get a >>> report what every thread was calling at the time of crash. >>> >>> >>> Regards, >>> Sergej >>> >>> On 28 May 2018, at 23:46, Igor Clark wrote: >>>> >>>> Hi folks, hope all well, >>>> >>>> I have a NIF which very occasionally segfaults, intermittently and >>>> apparently unpredictably, bringing down the VM. I've spent a bunch of time >>>> tracing allocation and dereferencing problems in my NIF code, and I've got >>>> rid of what seems like 99%+ of the problems - but it still occasionally >>>> happens, and I'm having trouble tracing further, because the crash logs >>>> show the crashed threads as doing things like these: (each one taken from a >>>> separate log where it's the only crashed thread) >>>> >>>> >>>> Thread 40 Crashed:: 8_scheduler >>>>> 0 beam.smp 0x000000001c19980b process_main >>>>> + 1570 >>>>> >>>>> Thread 5 Crashed:: 3_scheduler >>>>> 0 beam.smp 0x000000001c01d80b process_main >>>>> + 1570 >>>>> >>>>> Thread 7 Crashed:: 5_scheduler >>>>> 0 beam.smp 0x000000001baff0b8 >>>>> lists_member_2 + 63 >>>>> >>>>> Thread 3 Crashed:: 1_scheduler >>>>> 0 beam.smp 0x000000001d4b780b process_main >>>>> + 1570 >>>>> >>>>> Thread 5 Crashed:: 3_scheduler >>>>> 0 beam.smp 0x000000001fcf280b process_main >>>>> + 1570 >>>>> >>>>> Thread 6 Crashed:: 4_scheduler >>>>> 0 beam.smp 0x000000001ae290b8 >>>>> lists_member_2 + 63 >>>>> >>>> >>>> I'm very confident that the problems are in my code, not in the >>>> scheduler ;-) But without more detail, I don't know how to trace where >>>> they're happening. When they do, there are sometimes other threads doing >>>> things in my code (maybe 20% of the time) - but mostly not, and on the >>>> occasions when they are, I've been unable to see what the problem might be >>>> on the lines referenced. >>>> >>>> It seems like it's some kind of cross-thread data access issue, but I >>>> don't know how to track it down. >>>> >>>> Some more context about what's going on. My NIF load() function starts >>>> a thread which passes a callback function to a library that talks to some >>>> hardware, which calls the callback when it has a message. It's a separate >>>> thread because the library only calls back to the thread that initialized >>>> it; when I ran it directly in NIF load(), it didn't call back, but in the >>>> VM-managed thread, it works as expected. The thread sits and waits for >>>> stuff to happen, and callbacks come when they should. >>>> >>>> I use enif_thread_create/enif_thread_opts_create to start the thread, >>>> and use enif_alloc/enif_free everywhere. I keep a static pointer in the NIF >>>> to a couple of members of the state struct, as that seems the only way to >>>> reference them in the callback function. The struct is kept in NIF private >>>> data: I pass **priv from load() to the thread_main function, allocate the >>>> state struct using enif_alloc in thread_main, and set priv pointing to the >>>> state struct, also in the thread. Other NIF functions do access members of >>>> the state struct, but only ever through enif_priv_data( env ). >>>> >>>> The vast majority of the time it all works perfectly, humming along >>>> very nicely, but every now and then, without any real pattern I can see, it >>>> just segfaults and the VM comes down. It's only happened 3 times in the >>>> last 20+ hours of working on the app, testing & running all the while, >>>> doing VM starts, stops, code reloads, etc. But when it happens, it's kind >>>> of a showstopper, and I'd really like to nail it down. >>>> >>>> This is all happening in Erlang 20.3.4 on MacOS 10.12.6 / Apple LLVM >>>> version 9.0.0 (clang-900.0.38). >>>> >>>> Any ideas on how/where to look next to try to track this down? Hope >>>> it's not something structural in the above which just won't work. >>>> >>>> Cheers, >>>> Igor >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gomoripeti@REDACTED Tue May 29 15:09:04 2018 From: gomoripeti@REDACTED (=?UTF-8?B?UGV0aSBHw7Ztw7ZyaQ==?=) Date: Tue, 29 May 2018 15:09:04 +0200 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: since OTP 20 the *-emu_type* flag might also work eg.: erl -emu_type debug and you can put it in the vm.args file too On Tue, May 29, 2018 at 2:45 PM, Lukas Larsson wrote: > I don't know how to make rebar3 run the debug emulator, but a quick and > dirty trick that I do when all else fails is to copy the beam.debug.smp > file over the beam.smp file. > > You probably also have to copy the erl_child_setup.debug file, that file > should however have the .debug suffix remaining. So: > > cp bin/`erts/autoconf/config.guess`/beam.debug.smp > path/to/release/erts-v.s.n/bin/beam.smp > cp bin/`erts/autoconf/config.guess`/erl_child_setup.debug > path/to/release/erts-v.s.n/bin/ > > > On Tue, May 29, 2018 at 1:30 PM, Igor Clark wrote: > >> Thanks very much Lukas, I think the debug emulator could be what I'm >> looking for. The NIF only sometimes crashes on lists:member/2 - those log >> lines are all from different crashes (there's only one crashed thread each >> time), and sometimes it just crashes on process_main. So I think I might >> need the debug emulator to trace further. >> >> However I have a lot to learn about how to integrate C tooling with >> something so complex. When I run the debug emulator, does it just show more >> detailed info in stack traces, or will I need to attach gdb/lldb etc to >> find out what's going on? Is there any more info on how to set this all up? >> >> Also, not 100% sure how to run it, as I run my app with "rebar3 shell" >> from a release layout during development, or the same inside the >> NIF-specific app when trying to track problems down there. The doc you >> linked says: >> >> To start the debug enabled runtime system execute: >> >> $ $ERL_TOP/bin/cerl -debug >> >> >> I realise these are more rebar3 than erlang questions, but I can't find >> much in the rebar3 docs about them: >> >> - How should I specify that rebar3 should run "cerl" instead of "erl" ? >> >> - Should I just add "-debug" in my "config/vm.args" or is there another >> way to do this? >> >> Thank you for your help! >> i >> >> >> On 29/05/2018 11:30, Lukas Larsson wrote: >> >> Have you tried to run your code in a debug emulator? https://github.com/e >> rlang/otp/blob/master/HOWTO/INSTALL.md#how-to-build-a-debug- >> enabled-erlang-runtime-system >> >> Since it seems to be segfaulting in lists:member/2, I would guess that >> your nif somehow builds an invalid list that later is used by >> lists:member/2. >> >> On Tue, May 29, 2018 at 11:04 AM, Igor Clark >> wrote: >> >>> Thanks Sergej - that's where I got the thread reports I pasted in below, >>> from e.g. 'beam.smp_2018-05-28-212735_Igor-Clarks-iMac.crash'. >>> >>> Each log says the only crashed thread was a scheduler thread, for >>> example "8_scheduler" running "process_main" in the case of the first one >>> below. This is how I tracked down a bunch of errors in my own code, but the >>> only ones that still happen are in the scheduler, according to the Console >>> crash logs. >>> >>> The thing is, it seems really unlikely that a VM running my NIF code >>> would just happen to be crashing in the scheduler rather than my code(!) - >>> so that's what I'm trying to work out, how to find out what's actually >>> going on, given that the log tells me the crashed thread is running >>> "process_main" or 'lists_member_2'. >>> >>> Any suggestions welcome! >>> >>> Cheers, >>> Igor >>> >>> >>> On 29/05/2018 04:16, Sergej Jure?ko wrote: >>> >>>> On macOS there is a quick way to get a stack trace if you compiled with >>>> debug symbols. >>>> Open /Applications/Utilities/Console >>>> Go to: User Reports >>>> >>>> You will see beam.smp in there if it crashed. Click on it and you get a >>>> report what every thread was calling at the time of crash. >>>> >>>> >>>> Regards, >>>> Sergej >>>> >>>> On 28 May 2018, at 23:46, Igor Clark wrote: >>>>> >>>>> Hi folks, hope all well, >>>>> >>>>> I have a NIF which very occasionally segfaults, intermittently and >>>>> apparently unpredictably, bringing down the VM. I've spent a bunch of time >>>>> tracing allocation and dereferencing problems in my NIF code, and I've got >>>>> rid of what seems like 99%+ of the problems - but it still occasionally >>>>> happens, and I'm having trouble tracing further, because the crash logs >>>>> show the crashed threads as doing things like these: (each one taken from a >>>>> separate log where it's the only crashed thread) >>>>> >>>>> >>>>> Thread 40 Crashed:: 8_scheduler >>>>>> 0 beam.smp 0x000000001c19980b process_main >>>>>> + 1570 >>>>>> >>>>>> Thread 5 Crashed:: 3_scheduler >>>>>> 0 beam.smp 0x000000001c01d80b process_main >>>>>> + 1570 >>>>>> >>>>>> Thread 7 Crashed:: 5_scheduler >>>>>> 0 beam.smp 0x000000001baff0b8 >>>>>> lists_member_2 + 63 >>>>>> >>>>>> Thread 3 Crashed:: 1_scheduler >>>>>> 0 beam.smp 0x000000001d4b780b process_main >>>>>> + 1570 >>>>>> >>>>>> Thread 5 Crashed:: 3_scheduler >>>>>> 0 beam.smp 0x000000001fcf280b process_main >>>>>> + 1570 >>>>>> >>>>>> Thread 6 Crashed:: 4_scheduler >>>>>> 0 beam.smp 0x000000001ae290b8 >>>>>> lists_member_2 + 63 >>>>>> >>>>> >>>>> I'm very confident that the problems are in my code, not in the >>>>> scheduler ;-) But without more detail, I don't know how to trace where >>>>> they're happening. When they do, there are sometimes other threads doing >>>>> things in my code (maybe 20% of the time) - but mostly not, and on the >>>>> occasions when they are, I've been unable to see what the problem might be >>>>> on the lines referenced. >>>>> >>>>> It seems like it's some kind of cross-thread data access issue, but I >>>>> don't know how to track it down. >>>>> >>>>> Some more context about what's going on. My NIF load() function starts >>>>> a thread which passes a callback function to a library that talks to some >>>>> hardware, which calls the callback when it has a message. It's a separate >>>>> thread because the library only calls back to the thread that initialized >>>>> it; when I ran it directly in NIF load(), it didn't call back, but in the >>>>> VM-managed thread, it works as expected. The thread sits and waits for >>>>> stuff to happen, and callbacks come when they should. >>>>> >>>>> I use enif_thread_create/enif_thread_opts_create to start the thread, >>>>> and use enif_alloc/enif_free everywhere. I keep a static pointer in the NIF >>>>> to a couple of members of the state struct, as that seems the only way to >>>>> reference them in the callback function. The struct is kept in NIF private >>>>> data: I pass **priv from load() to the thread_main function, allocate the >>>>> state struct using enif_alloc in thread_main, and set priv pointing to the >>>>> state struct, also in the thread. Other NIF functions do access members of >>>>> the state struct, but only ever through enif_priv_data( env ). >>>>> >>>>> The vast majority of the time it all works perfectly, humming along >>>>> very nicely, but every now and then, without any real pattern I can see, it >>>>> just segfaults and the VM comes down. It's only happened 3 times in the >>>>> last 20+ hours of working on the app, testing & running all the while, >>>>> doing VM starts, stops, code reloads, etc. But when it happens, it's kind >>>>> of a showstopper, and I'd really like to nail it down. >>>>> >>>>> This is all happening in Erlang 20.3.4 on MacOS 10.12.6 / Apple LLVM >>>>> version 9.0.0 (clang-900.0.38). >>>>> >>>>> Any ideas on how/where to look next to try to track this down? Hope >>>>> it's not something structural in the above which just won't work. >>>>> >>>>> Cheers, >>>>> Igor >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott_ribe@REDACTED Tue May 29 15:45:48 2018 From: scott_ribe@REDACTED (Scott Ribe) Date: Tue, 29 May 2018 07:45:48 -0600 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: Have you considered opening multiple sockets from multiple processes on the same port? It seems that your major problem was with high CPU usage so I'm not sure that technique would help--unless you were maxing out only a single core. On BSD and recent Linux kernels this will result in incoming packets being load-balanced across the sockets. It's not directly supported in Erlang, but the "raw" option lets you do it. I've got it working on OS X, though I've not yet load tested it. I haven't had the chance to verify the Linux branch, but here it is (the following is Elixir, not Erlang, but close enough): def reuseport_option() do case :os.type() do {:unix, name} -> cond do name in [:darwin, :freebsd, :openbsd, :netbsd] -> [{:raw, 0xffff, 0x0200, <<1::native-integer-size(32)>>} ] name in [:linux] -> [{:raw, 1, 15, <<1::native-integer-size(32)>>} ] false -> [] end _ -> [] end end Just append that result to your options. (Of course, if you run on a platform which is not supported here, if you try to open second and subsequent sockets you'll get errors, so use the [] return as a "can't do this" flag if there's any chance you'll run on such a platform.) -- Scott Ribe scott_ribe@REDACTED https://www.linkedin.com/in/scottribe/ > On May 26, 2018, at 12:17 AM, Lukas Larsson wrote: > > > > On Sat, May 26, 2018 at 3:38 AM, Max Lapshin wrote: > Ok, will check with reducing buffer. > > We have put 2 MB and even 16 MB because without it, we got packet drops > > You could also try raising the sbct limit to over 2 MB. i.e. > > %% Raise sbct limit > +MBsbct 4096 +MBlmbcs 20480 > > or > > %% lower user-space buffer > Common = [binary,{reuseaddr,true},{buffer,256*1024},{recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Tue May 29 15:57:31 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 29 May 2018 09:57:31 -0400 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: On Tue, May 29, 2018 at 7:30 AM, Igor Clark wrote: > > - How should I specify that rebar3 should run "cerl" instead of "erl" ? > You can't, especially if you're using 'rebar3 shell', where all we can do is boot a shell within an escript. Do try to debug from a release. You can probably do something like cerl -boot _build/prod/rel//release// -config _build/prod/rel//release//sys or if possible: _build/prod/rel//erts-/bin/erl -boot _build/prod/rel/release// -config _build/prod/rel/release//sys -emu_type debug These two commands will ignore the contents of your vm.args but they let you call the underlying mechanism to boot the VM with the arguments you need without the nodetool wrappers that relx ships. -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Tue May 29 16:08:02 2018 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 29 May 2018 15:08:02 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: Thanks Lukas and Peti, that's great. "erl -emu_type debug" definitely works - I haven't made the debug build yet but I get "erlexec: The emulator '/usr/local/Cellar/erlang/20.3.4/lib/erlang/erts-9.3/bin/beam.debug.smp' does not exist", which is what I want. I'll get onto the debug build and see what I can find out. In case anyone else wants to use that in rebar3 shell, I found http://www.rebar3.org/v3.0/discuss/5745fb105528582000dfb47f which shows you can set ERL_FLAGS to just set -emu_type directly, or specify vm.args so you can then set it in there, e.g.: > ERL_FLAGS=" -args_file config/vm.args -config config/sys.config" > rebar3 shell Cheers, Igor On 29/05/2018 14:09, Peti G?m?ri wrote: > since OTP 20 the *-emu_type*?flag might also work eg.: > ? erl -emu_type debug > > and you can put it in the vm.args file too > > On Tue, May 29, 2018 at 2:45 PM, Lukas Larsson > wrote: > > I don't know how to make rebar3 run the debug emulator, but a > quick and dirty trick that I do when all else fails is to copy the > beam.debug.smp file over the beam.smp file. > > You probably also have to copy the erl_child_setup.debug file, > that file should however have the .debug suffix remaining. So: > > cp bin/`erts/autoconf/config.guess`/beam.debug.smp > path/to/release/erts-v.s.n/bin/beam.smp > cp bin/`erts/autoconf/config.guess`/erl_child_setup.debug > path/to/release/erts-v.s.n/bin/ > > > On Tue, May 29, 2018 at 1:30 PM, Igor Clark > wrote: > > Thanks very much Lukas, I think the debug emulator could be > what I'm looking for. The NIF only sometimes crashes on > lists:member/2 - those log lines are all from different > crashes (there's only one crashed thread each time), and > sometimes it just crashes on process_main. So I think I might > need the debug emulator to trace further. > > However I have a lot to learn about how to integrate C tooling > with something so complex. When I run the debug emulator, does > it just show more detailed info in stack traces, or will I > need to attach gdb/lldb etc to find out what's going on? Is > there any more info on how to set this all up? > > Also, not 100% sure how to run it, as I run my app with > "rebar3 shell" from a release layout during development, or > the same inside the NIF-specific app when trying to track > problems down there. The doc you linked says: > >> To start the debug enabled runtime system execute: >> >> |$ $ERL_TOP/bin/cerl -debug| > > I realise these are more rebar3 than erlang questions, but I > can't find much in the rebar3 docs about them: > > - How should I specify that rebar3 should run "cerl" instead > of "erl" ? > > - Should I just add "-debug" in my "config/vm.args" or is > there another way to do this? > > Thank you for your help! > i > > > On 29/05/2018 11:30, Lukas Larsson wrote: >> Have you tried to run your code in a debug emulator? >> https://github.com/erlang/otp/blob/master/HOWTO/INSTALL.md#how-to-build-a-debug-enabled-erlang-runtime-system >> >> >> >> Since it seems to be segfaulting in lists:member/2, I would >> guess that your nif somehow builds an invalid list that later >> is used by lists:member/2. >> >> On Tue, May 29, 2018 at 11:04 AM, Igor Clark >> > wrote: >> >> Thanks Sergej - that's where I got the thread reports I >> pasted in below, from e.g. >> 'beam.smp_2018-05-28-212735_Igor-Clarks-iMac.crash'. >> >> Each log says the only crashed thread was a scheduler >> thread, for example "8_scheduler" running "process_main" >> in the case of the first one below. This is how I tracked >> down a bunch of errors in my own code, but the only ones >> that still happen are in the scheduler, according to the >> Console crash logs. >> >> The thing is, it seems really unlikely that a VM running >> my NIF code would just happen to be crashing in the >> scheduler rather than my code(!) - so that's what I'm >> trying to work out, how to find out what's actually going >> on, given that the log tells me the crashed thread is >> running "process_main" or 'lists_member_2'. >> >> Any suggestions welcome! >> >> Cheers, >> Igor >> >> >> On 29/05/2018 04:16, Sergej Jure?ko wrote: >> >> On macOS there is a quick way to get a stack trace if >> you compiled with debug symbols. >> Open /Applications/Utilities/Console >> Go to: User Reports >> >> You will see beam.smp in there if it crashed. Click >> on it and you get a report what every thread was >> calling at the time of crash. >> >> >> Regards, >> Sergej >> >> On 28 May 2018, at 23:46, Igor Clark >> > > wrote: >> >> Hi folks, hope all well, >> >> I have a NIF which very occasionally segfaults, >> intermittently and apparently unpredictably, >> bringing down the VM. I've spent a bunch of time >> tracing allocation and dereferencing problems in >> my NIF code, and I've got rid of what seems like >> 99%+ of the problems - but it still occasionally >> happens, and I'm having trouble tracing further, >> because the crash logs show the crashed threads >> as doing things like these: (each one taken from >> a separate log where it's the only crashed thread) >> >> >> Thread 40 Crashed:: 8_scheduler >> 0? ?beam.smp 0x000000001c19980b process_main >> + 1570 >> >> Thread 5 Crashed:: 3_scheduler >> 0? ?beam.smp 0x000000001c01d80b process_main >> + 1570 >> >> Thread 7 Crashed:: 5_scheduler >> 0? ?beam.smp 0x000000001baff0b8 >> lists_member_2 + 63 >> >> Thread 3 Crashed:: 1_scheduler >> 0? ?beam.smp 0x000000001d4b780b process_main >> + 1570 >> >> Thread 5 Crashed:: 3_scheduler >> 0? ?beam.smp 0x000000001fcf280b process_main >> + 1570 >> >> Thread 6 Crashed:: 4_scheduler >> 0? ?beam.smp 0x000000001ae290b8 >> lists_member_2 + 63 >> >> >> I'm very confident that the problems are in my >> code, not in the scheduler ;-) But without more >> detail, I don't know how to trace where they're >> happening. When they do, there are sometimes >> other threads doing things in my code (maybe 20% >> of the time) - but mostly not, and on the >> occasions when they are, I've been unable to see >> what the problem might be on the lines referenced. >> >> It seems like it's some kind of cross-thread data >> access issue, but I don't know how to track it down. >> >> Some more context about what's going on. My NIF >> load() function starts a thread which passes a >> callback function to a library that talks to some >> hardware, which calls the callback when it has a >> message. It's a separate thread because the >> library only calls back to the thread that >> initialized it; when I ran it directly in NIF >> load(), it didn't call back, but in the >> VM-managed thread, it works as expected. The >> thread sits and waits for stuff to happen, and >> callbacks come when they should. >> >> I use enif_thread_create/enif_thread_opts_create >> to start the thread, and use enif_alloc/enif_free >> everywhere. I keep a static pointer in the NIF to >> a couple of members of the state struct, as that >> seems the only way to reference them in the >> callback function. The struct is kept in NIF >> private data: I pass **priv from load() to the >> thread_main function, allocate the state struct >> using enif_alloc in thread_main, and set priv >> pointing to the state struct, also in the thread. >> Other NIF functions do access members of the >> state struct, but only ever through >> enif_priv_data( env ). >> >> The vast majority of the time it all works >> perfectly, humming along very nicely, but every >> now and then, without any real pattern I can see, >> it just segfaults and the VM comes down. It's >> only happened 3 times in the last 20+ hours of >> working on the app, testing & running all the >> while, doing VM starts, stops, code reloads, etc. >> But when it happens, it's kind of a showstopper, >> and I'd really like to nail it down. >> >> This is all happening in Erlang 20.3.4 on MacOS >> 10.12.6 / Apple LLVM version 9.0.0 (clang-900.0.38). >> >> Any ideas on how/where to look next to try to >> track this down? Hope it's not something >> structural in the above which just won't work. >> >> Cheers, >> Igor >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Tue May 29 16:23:49 2018 From: t@REDACTED (Tristan Sloughter) Date: Tue, 29 May 2018 08:23:49 -0600 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> Message-ID: <1527603829.2691628.1389151640.4A610BF9@webmail.messagingengine.com> If you need cerl this is true. But if you just need `-emu_type debug` then if you do `rebar3 local install` and follow the instructions to update your $PATH you can do: REBAR3_ERL_ARGS="-emu_type debug" rebar3 shell On Tue, May 29, 2018, at 7:57 AM, Fred Hebert wrote: > > On Tue, May 29, 2018 at 7:30 AM, Igor Clark > wrote:>> >> - How should I specify that rebar3 should run "cerl" instead of >> "erl" ?> > You can't, especially if you're using 'rebar3 shell', where all we can > do is boot a shell within an escript. Do try to debug from a release. > You can probably do something like> > cerl -boot _build/prod/rel//release// - > config _build/prod/rel//release//sys> > or if possible: > > _build/prod/rel//erts-/bin/erl -boot > _build/prod/rel/release// -config > _build/prod/rel/release//sys -emu_type debug> > These two commands will ignore the contents of your vm.args but they > let you call the underlying mechanism to boot the VM with the > arguments you need without the nodetool wrappers that relx ships.> _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Tue May 29 16:33:56 2018 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 29 May 2018 15:33:56 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> Message-ID: Thanks Scott, that sounds very much like what's going on and using libgmalloc sounds like it's just what I want. From what I can tell from its docs, I could either do that in Xcode, or using gdb, right? I have no clue how I'd go about running the BEAM inside Xcode, especially booting my application and NIF code, and I've got a lot to learn about how to connect any of this with rebar3 and releases. Do you run BEAM in Xcode or directly with gdb/lldb? How do the people debug the emulator versions with applications loaded? On 29/05/2018 13:20, Scott Ribe wrote: > What's most likely going on is that your NIF is corrupting memory that the scheduler uses. These kinds of corruptions are difficult to track down, because the crash can happen long after the bug. > > You should be running with all memory debugging aids turned on--this can get you a crash closer to the actual bug, if you're lucky maybe even at the bug. > > https://developer.apple.com/library/content/documentation/Performance/Conceptual/ManagingMemory/Articles/MallocDebug.html > > https://developer.apple.com/library/content/technotes/tn2124/_index.html > > -- > Scott Ribe > scott_ribe@REDACTED > https://www.linkedin.com/in/scottribe/ > > > >> On May 29, 2018, at 3:04 AM, Igor Clark wrote: >> >> Thanks Sergej - that's where I got the thread reports I pasted in below, from e.g. 'beam.smp_2018-05-28-212735_Igor-Clarks-iMac.crash'. >> >> Each log says the only crashed thread was a scheduler thread, for example "8_scheduler" running "process_main" in the case of the first one below. This is how I tracked down a bunch of errors in my own code, but the only ones that still happen are in the scheduler, according to the Console crash logs. >> >> The thing is, it seems really unlikely that a VM running my NIF code would just happen to be crashing in the scheduler rather than my code(!) - so that's what I'm trying to work out, how to find out what's actually going on, given that the log tells me the crashed thread is running "process_main" or 'lists_member_2'. >> >> Any suggestions welcome! >> >> Cheers, >> Igor >> >> On 29/05/2018 04:16, Sergej Jure?ko wrote: From scott_ribe@REDACTED Tue May 29 16:42:34 2018 From: scott_ribe@REDACTED (Scott Ribe) Date: Tue, 29 May 2018 08:42:34 -0600 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> Message-ID: <7342393D-9CAA-42C8-AD09-665423BC91D2@elevated-dev.com> > On May 29, 2018, at 8:33 AM, Igor Clark wrote: > > From what I can tell from its docs, I could either do that in Xcode, or using gdb, right? I have no clue how I'd go about running the BEAM inside Xcode, especially booting my application and NIF code, and I've got a lot to learn about how to connect any of this with rebar3 and releases. Do you run BEAM in Xcode or directly with gdb/lldb? Actually, all you have to do is export those environment variables in whatever environment you launch, not just Xcode or gdb. At launch, they barf some descriptive info to stdout (or stderr?), so if you can figure out where the beam sends that and monitor it, you should see confirmation that they're in effect: SHR-MacBook-Pro:~ sribe$ export MallocStackLogging=1 SHR-MacBook-Pro:~ sribe$ ls ls(32181,0x7fff88acb380) malloc: stack logs being written into /tmp/stack-logs.32181.104a26000.ls.6dB2BT.index ls(32181,0x7fff88acb380) malloc: recording malloc and VM allocation stacks to disk using standard recorder Applications Dropbox Public eclipse fsaclctl Desktop Library Sites eclipse-workspace perf_demo Desktop Folder Movies TheFindByContentFolder elixir-basics-workshop pgdata Documents Music Trash elixir-fire-brigade-workshop Downloads Pictures _rvm elixirconf_workshop ls(32181,0x7fff88acb380) malloc: stack logs deleted from /tmp/stack-logs.32181.104a26000.ls.6dB2BT.index SHR-MacBook-Pro:~ sribe$ Barring that, you could always insert a deliberate overrun in your NIF's initialization and check that it immediately crashes with an access violation. -- Scott Ribe scott_ribe@REDACTED https://www.linkedin.com/in/scottribe/ From igor.clark@REDACTED Tue May 29 17:16:51 2018 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 29 May 2018 16:16:51 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: <7342393D-9CAA-42C8-AD09-665423BC91D2@elevated-dev.com> References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> <7342393D-9CAA-42C8-AD09-665423BC91D2@elevated-dev.com> Message-ID: <9cdd425a-e8a0-c37e-efe4-932f64417925@gmail.com> OK, great, thanks Scott! Combining this with Fred's help on running the release with custom args, I have this: > DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib MallocGuardEdges=1 > MallocStackLogging=1 _build/prod/rel//erts-9.3/bin/erl -boot > _build/prod/rel//releases/0.1.0/ -config > _build/prod/rel//releases/0.1.0/sys.config which runs and gives this: > beam.smp(62326,0x7fffa52133c0) malloc: stack logs being written into > /tmp/stack-logs.62326.1f5a2000.beam.smp.QWSLfS.index which seems good, the file is present and gets updated. So, do I have this right: the point of the Guard Malloc is to make the crash happen at the time of allocation, rather than delayed until something trying to access it triggers the segfault; so if I get a crash while running like this, I should be able to just check in the Console debug log, and the stack trace should show where the bug actually is? Or does this only work if I run the entire process through gdb or lldb? cheers, i On 29/05/2018 15:42, Scott Ribe wrote: >> On May 29, 2018, at 8:33 AM, Igor Clark wrote: >> >> From what I can tell from its docs, I could either do that in Xcode, or using gdb, right? I have no clue how I'd go about running the BEAM inside Xcode, especially booting my application and NIF code, and I've got a lot to learn about how to connect any of this with rebar3 and releases. Do you run BEAM in Xcode or directly with gdb/lldb? > Actually, all you have to do is export those environment variables in whatever environment you launch, not just Xcode or gdb. > > At launch, they barf some descriptive info to stdout (or stderr?), so if you can figure out where the beam sends that and monitor it, you should see confirmation that they're in effect: > > > SHR-MacBook-Pro:~ sribe$ export MallocStackLogging=1 > SHR-MacBook-Pro:~ sribe$ ls > ls(32181,0x7fff88acb380) malloc: stack logs being written into /tmp/stack-logs.32181.104a26000.ls.6dB2BT.index > ls(32181,0x7fff88acb380) malloc: recording malloc and VM allocation stacks to disk using standard recorder > Applications Dropbox Public eclipse fsaclctl > Desktop Library Sites eclipse-workspace perf_demo > Desktop Folder Movies TheFindByContentFolder elixir-basics-workshop pgdata > Documents Music Trash elixir-fire-brigade-workshop > Downloads Pictures _rvm elixirconf_workshop > ls(32181,0x7fff88acb380) malloc: stack logs deleted from /tmp/stack-logs.32181.104a26000.ls.6dB2BT.index > SHR-MacBook-Pro:~ sribe$ > > > Barring that, you could always insert a deliberate overrun in your NIF's initialization and check that it immediately crashes with an access violation. > > -- > Scott Ribe > scott_ribe@REDACTED > https://www.linkedin.com/in/scottribe/ > > > > From scott_ribe@REDACTED Tue May 29 17:31:48 2018 From: scott_ribe@REDACTED (Scott Ribe) Date: Tue, 29 May 2018 09:31:48 -0600 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: <9cdd425a-e8a0-c37e-efe4-932f64417925@gmail.com> References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> <7342393D-9CAA-42C8-AD09-665423BC91D2@elevated-dev.com> <9cdd425a-e8a0-c37e-efe4-932f64417925@gmail.com> Message-ID: <51C26C64-84A8-4B9C-882E-9D52C019172F@elevated-dev.com> > On May 29, 2018, at 9:16 AM, Igor Clark wrote: > > So, do I have this right: the point of the Guard Malloc is to make the crash happen at the time of allocation, rather than delayed until something trying to access it triggers the segfault; so if I get a crash while running like this, I should be able to just check in the Console debug log, and the stack trace should show where the bug actually is? At the time of the illegal access, not the allocation. Yes, that's the point, you get a stack trace showing you illegal access. However, the BEAM allocator will reduce its effectiveness. When you malloc in your C code, you get a block set up such that accessing just past it (or potentially before it) will cause an immediate crash. When you free it, it's then set up such that accessing will cause an immediate crash. But if you use Erlang's allocation routines, Erlang may malloc a bigger block with those protections, then hand out multiple suballocations, and access beyond the end of one of those can simply corrupt the next one without crashing at that point. You should also be using MallocScribble & MallocPreScribble. From gordeev.vladimir.v@REDACTED Tue May 29 22:08:11 2018 From: gordeev.vladimir.v@REDACTED (Vladimir Gordeev) Date: Tue, 29 May 2018 23:08:11 +0300 Subject: [erlang-questions] Fwd: [ANN] Batiscaph seeing tool v0.1.1: visualization of the processes on the map In-Reply-To: References: Message-ID: On Tue, May 29, 2018 at 10:02 PM, Olivier Girondel wrote: > > > Amazing work ! Will definitely try it even if it's not ready for > production > > Hope your project will grow ! > Thanks! On Tue, May 29, 2018 at 10:09 PM, Olivier Girondel wrote: > > > Is the typo in the name on purpose ? > > The diving tool is bathyscaph Yeah, that was on purpose. Makes it easier to google it later. -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Tue May 29 22:15:47 2018 From: igor.clark@REDACTED (Igor Clark) Date: Tue, 29 May 2018 21:15:47 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: <51C26C64-84A8-4B9C-882E-9D52C019172F@elevated-dev.com> References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> <7342393D-9CAA-42C8-AD09-665423BC91D2@elevated-dev.com> <9cdd425a-e8a0-c37e-efe4-932f64417925@gmail.com> <51C26C64-84A8-4B9C-882E-9D52C019172F@elevated-dev.com> Message-ID: <3ac44ee7-1047-164c-08b3-b13fe723806c@gmail.com> OK. Thanks very much Scott. I've got all this working using both those extra options, and it does seem to make the NIF crash a lot sooner than previously, which is great. But I'm still only seeing "process_main" in the crashed thread, so I'm not much closer to knowing where the illegal access is. I wonder if it's in lots of places because of what I'm doing with the callback and the thread. I hope not. I'll do some more digging, and tomorrow I'll try out a debug emulator build as well. Thanks very much for helping me get this far! On 29/05/2018 16:31, Scott Ribe wrote: >> On May 29, 2018, at 9:16 AM, Igor Clark wrote: >> >> So, do I have this right: the point of the Guard Malloc is to make the crash happen at the time of allocation, rather than delayed until something trying to access it triggers the segfault; so if I get a crash while running like this, I should be able to just check in the Console debug log, and the stack trace should show where the bug actually is? > At the time of the illegal access, not the allocation. Yes, that's the point, you get a stack trace showing you illegal access. > > However, the BEAM allocator will reduce its effectiveness. When you malloc in your C code, you get a block set up such that accessing just past it (or potentially before it) will cause an immediate crash. When you free it, it's then set up such that accessing will cause an immediate crash. But if you use Erlang's allocation routines, Erlang may malloc a bigger block with those protections, then hand out multiple suballocations, and access beyond the end of one of those can simply corrupt the next one without crashing at that point. > > You should also be using MallocScribble & MallocPreScribble. > > > From dmorneau@REDACTED Wed May 30 00:58:56 2018 From: dmorneau@REDACTED (Dominic Morneau) Date: Wed, 30 May 2018 07:58:56 +0900 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: <3ac44ee7-1047-164c-08b3-b13fe723806c@gmail.com> References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> <7342393D-9CAA-42C8-AD09-665423BC91D2@elevated-dev.com> <9cdd425a-e8a0-c37e-efe4-932f64417925@gmail.com> <51C26C64-84A8-4B9C-882E-9D52C019172F@elevated-dev.com> <3ac44ee7-1047-164c-08b3-b13fe723806c@gmail.com> Message-ID: Can you give it a try with "+Mea min" in erl options? This should make Erlang fall back to malloc for all allocators, hopefully making guard malloc more effective. Dominic 2018?5?30?(?) 5:15 Igor Clark : > OK. Thanks very much Scott. I've got all this working using both those > extra options, and it does seem to make the NIF crash a lot sooner than > previously, which is great. But I'm still only seeing "process_main" in > the crashed thread, so I'm not much closer to knowing where the illegal > access is. I wonder if it's in lots of places because of what I'm doing > with the callback and the thread. I hope not. > > I'll do some more digging, and tomorrow I'll try out a debug emulator > build as well. > > Thanks very much for helping me get this far! > > On 29/05/2018 16:31, Scott Ribe wrote: > >> On May 29, 2018, at 9:16 AM, Igor Clark wrote: > >> > >> So, do I have this right: the point of the Guard Malloc is to make the > crash happen at the time of allocation, rather than delayed until something > trying to access it triggers the segfault; so if I get a crash while > running like this, I should be able to just check in the Console debug log, > and the stack trace should show where the bug actually is? > > At the time of the illegal access, not the allocation. Yes, that's the > point, you get a stack trace showing you illegal access. > > > > However, the BEAM allocator will reduce its effectiveness. When you > malloc in your C code, you get a block set up such that accessing just past > it (or potentially before it) will cause an immediate crash. When you free > it, it's then set up such that accessing will cause an immediate crash. But > if you use Erlang's allocation routines, Erlang may malloc a bigger block > with those protections, then hand out multiple suballocations, and access > beyond the end of one of those can simply corrupt the next one without > crashing at that point. > > > > You should also be using MallocScribble & MallocPreScribble. > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlangsiri@REDACTED Wed May 30 10:41:38 2018 From: erlangsiri@REDACTED (Siri Hansen) Date: Wed, 30 May 2018 10:41:38 +0200 Subject: [erlang-questions] Logger error with OTP 21.0-rc1 In-Reply-To: References: Message-ID: Hi again - sorry for the slow follow-up! The problem is that start script/boot built with older releases start the error_logger instead of logger. Good to see you solved it - thanks for letting me know! /siri 2018-05-29 14:04 GMT+02:00 S?bastien Merle : > After some unsuccessful investigation I re-read your message I realized I > was still using 20.2 to release my application ! > This was the issue , everything seems to work fin when using 21.0-rc1 for > everything. > > Thank you very much. > > On Tue, May 29, 2018 at 12:45 PM, S?bastien Merle > wrote: > >> The release is built using rebar3: >> >> `{relx, [{release, {testproject, "0.1.0"}, [ testproject ]}]}.` >> >> and erlang is started on the board with: >> >> `erl.rtems -- -home . -pa . -root testproject -boot >> testproject/releases/0.1.0/testproject -kernel inetrc "./erl_inetrc"` >> >> Thank you for your help. >> >> Sebastien Merle, >> Peer Stritzinger GmbH. >> >> >> On Tue, May 29, 2018 at 11:21 AM, Siri Hansen >> wrote: >> >>> Hi Sebastien, >>> >>> which script/boot file do you use? Is it built with the new OTP version? >>> >>> Kind Regards >>> /siri@REDACTED >>> >>> 2018-05-28 23:42 GMT+02:00 S?bastien Merle : >>> >>>> Hi, >>>> >>>> I am porting the grisp software (grisp.org) to OTP 21.0-rc1. >>>> Everything build and deploy just fine but when running Erlang on the >>>> board the console is swamped with error like: >>>> >>>> `(no logger present) unexpected logger message: {log,error,"Error in >>>> process ~p with exit value:~n~p~n",[<0.39.0>,{badar >>>> g,[{ets,lookup,[logger,proc_lib],[]},{logger_config,allow,3, >>>> [{file,"logger_config.erl"},{...` >>>> >>>> I saw someone else complaining about this same error on the thread >>>> about 21.0-rc1 but the only response was to use a clean build, and I >>>> already built OTP from a fresh checkout. >>>> >>>> Any idea what could be the cause ? Maybe this is a know issue or the >>>> new logger API requires some configuration or service to be started >>>> explicitly ? >>>> >>>> Thank you very much. >>>> >>>> Sebastien Merle, >>>> Peer Stritzinger GmbH. >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik.x.nord@REDACTED Wed May 30 11:06:16 2018 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Wed, 30 May 2018 09:06:16 +0000 Subject: [erlang-questions] OTP 21 Release Candidate 2 available for testing Message-ID: OTP 21 Release Candidate 2 This is the second of two planned release candidates before the OTP 21 release. As we move closer to OTP 21 we aim to improve the quality and stability of the release candidate. So this candidate is mostly the same as the first one, but with bug-fixes and improvements to the new features mentioned before. The intention with this release is that you as users try it and give us feedback Erlang/OTP 21 is a new major release with new features, improvements as well as incompatibilities. Potential Incompatibilities * All Corba applications are now moved from the OTP repository * A new Corba repository will be created https://github.com/erlang * New applications ftp and tftp, moved from inets * ssl no longer supports 3_DES cipher suites or RSA-key exchange cipher suites by default * erlang:monitor on a primitive node (erl_interface, jinterface, etc) will no longer fail with badarg exception. Instead a monitor will be created, but it will only supervise the connection to the node. Highlights Erts: * Enhanced IO scalability * Support for usage of distribution controller processes for alternative transports, routing etc * compact instructions on 64bit systems for code below 4GB 20% less memory for loaded code * Rewrite of the efile-driver with NIFs and "Dirty schedulers" resulting in faster file operations * non-smp VM removed * link and monitor optimized for scalability * os:getenv/putenv now work on thread-safe emulation. No longer in sync with libc getenv(3). Manual synchronization will be needed. Compiler: * Misc compiler optimizations including contributions from the Elixir team resulting in 10% improvements in benchmarks * "Tuple calls" have been removed from the run-time system. * Code such as f({ok, Val}) -> {ok, Val} is now automatically rewritten to f({ok, Val} = Tuple) -> Tuple. this reduces code size, execution time, and removed GC pressure. * More information in stacktrace from a number of operators * erlang:get_stacktrace/0 deprecated to be replaced with try ... catch C:R:Stacktrace -> ... * Creation of small maps with literal keys optimized. Security: * DTLS support in SSL * Enhanced support for distribution over TLS * "unsecure" ciphers removed from defaults in SSL and SSH. * A new option value defined to facilitate implementing exec servers. Old option kept for compatibility, but now gives errors on stderror. Standard libraries: * New API for logging, logger * New uri_string module for parsing URIs according to "The standard" * New function lists:search(list,fun/1) -> {ok, Value} | false * Changed default behaviour of .erlang loading. escript, erlc, dialyzer and typer no longer load an .erlang at all. For more details see http://erlang.org/download/otp_src_21.0-rc2.readme Pre built versions for Windows can be fetched here: http://erlang.org/download/otp_win32_21.0-rc2.exe http://erlang.org/download/otp_win64_21.0-rc2.exe Online documentation can be browsed here: http://erlang.org/documentation/doc-10.0-rc2/doc The Erlang/OTP source can also be found at GitHub on the official Erlang repository, https://github.com/erlang/otp tag: OTP-21.0-rc2 Thank you for all your contributions! -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.clark@REDACTED Wed May 30 11:19:06 2018 From: igor.clark@REDACTED (Igor Clark) Date: Wed, 30 May 2018 10:19:06 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> <7342393D-9CAA-42C8-AD09-665423BC91D2@elevated-dev.com> <9cdd425a-e8a0-c37e-efe4-932f64417925@gmail.com> <51C26C64-84A8-4B9C-882E-9D52C019172F@elevated-dev.com> <3ac44ee7-1047-164c-08b3-b13fe723806c@gmail.com> Message-ID: Thanks Dominic - I don't want to count my chickens before they've hatched, but it looks like guard malloc has pointed me to at least some bugs even without that VM option. Even though I wasn't getting a line number in the stack trace, it was already seeming to make the NIF crash immediately and consistently, so I was able to use a ton of debug print statements to track down two problems that I hadn't been able to see before. (One was an enif_alloc() in the wrong place, and another seems to have been accessing a pointer from a function in a shared object file, oops.) No way would I have seen them without guard malloc showing me the way, it's a powerful tool :-) So I fixed those two, and right now the app is running as expected without crashes under guard malloc. I'm pretty sure that I'll come up against more illegal-access bugs over time, so I'm adding "+Mea min" to the list of options to use when I find the next one. Thank you. Thanks very much also to everyone who replied, particularly Scott for the guard malloc suggestion & help, and Fred & Tristan for the rebar3 tips so I could add the necessary CLI options and track down what was going on. I'm very glad to have been able to ask such experienced folks for advice, and to have learned about some *extremely* useful new stuff. Cheers, Igor On 29/05/2018 23:58, Dominic Morneau wrote: > Can you give it a try with "+Mea min" in erl options? This should make > Erlang fall back to malloc for all allocators, hopefully making guard > malloc more effective. > > Dominic > > 2018?5?30?(?) 5:15 Igor Clark >: > > OK. Thanks very much Scott. I've got all this working using both > those > extra options, and it does seem to make the NIF crash a lot sooner > than > previously, which is great. But I'm still only seeing > "process_main" in > the crashed thread, so I'm not much closer to knowing where the > illegal > access is. I wonder if it's in lots of places because of what I'm > doing > with the callback and the thread. I hope not. > > I'll do some more digging, and tomorrow I'll try out a debug emulator > build as well. > > Thanks very much for helping me get this far! > > On 29/05/2018 16:31, Scott Ribe wrote: > >> On May 29, 2018, at 9:16 AM, Igor Clark > wrote: > >> > >> So, do I have this right: the point of the Guard Malloc is to > make the crash happen at the time of allocation, rather than > delayed until something trying to access it triggers the segfault; > so if I get a crash while running like this, I should be able to > just check in the Console debug log, and the stack trace should > show where the bug actually is? > > At the time of the illegal access, not the allocation. Yes, > that's the point, you get a stack trace showing you illegal access. > > > > However, the BEAM allocator will reduce its effectiveness. When > you malloc in your C code, you get a block set up such that > accessing just past it (or potentially before it) will cause an > immediate crash. When you free it, it's then set up such that > accessing will cause an immediate crash. But if you use Erlang's > allocation routines, Erlang may malloc a bigger block with those > protections, then hand out multiple suballocations, and access > beyond the end of one of those can simply corrupt the next one > without crashing at that point. > > > > You should also be using MallocScribble & MallocPreScribble. > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From soumya.shankar.sardar@REDACTED Wed May 30 11:04:23 2018 From: soumya.shankar.sardar@REDACTED (Soumya Shankar Sardar) Date: Wed, 30 May 2018 09:04:23 +0000 Subject: [erlang-questions] Enabling SSL CRL revocation validation for secure URLS in Erlang Message-ID: Hi , Need some help on SSL CRL revocation validation enabled for HTTPS in Erlang code. 1) using httpc.requests to access the secure URL. 2) In the SSL options, we have made {verify:verify_peer} and {crl_check:peer}. 3) Also we have added the CRL file in local cache by ssl_crl_cache:insert(file). CRL file is downloaded from CDP via http. Questions 1) With above setup the CRL validation not failing for revoked URL. Any idea if the approach is wrong. we followed the erlang.org docs. 2) Also how we can extend this when there is a CDP[CRL distribution point] to get the dynamic CRL file. 3) And how to do above with CDP URL embedded in Server hello message in the SSL negotiation. It will be great to see a sample code with CRL validation in Erlang for SSL HTTP access. We are using Erlang/OTP 18.1. All comments are welcome :) Regards Soumya -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed May 30 20:44:59 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 30 May 2018 20:44:59 +0200 Subject: [erlang-questions] OTP 21 Release Candidate 2 available for testing In-Reply-To: References: Message-ID: <91cda262-20c7-5c50-a627-c87c900ca8f6@ninenines.eu> Looks good! The issues I had with rc1 are all gone and everything went fine on Alpine, Arch Linux, CentOS, Debian, FreeBSD, OSX and Ubuntu. I had a little difficulty building on OSX. When I had libjpeg 9c it asked for 8d, and when I switched to 8d it asked for 9c. But that's probably an issue with my environment, not an issue with OTP. Making both .dylib files available fixed it. Still putting it here just in case this rings a bell. On 05/30/2018 11:06 AM, Henrik Nord X wrote: > > OTP 21 Release Candidate 2 > > This is the second of two planned release candidates before the OTP 21 > release. > > As we move closer to OTP 21 we aim to improve the quality and stability > of the release candidate. > > So this candidate is mostly the same as the first one, but with > bug-fixes and improvements to the new features mentioned before. > > The intention with this release is that you as users try it and give us > feedback > > Erlang/OTP 21 is a new major release with new features, improvements as > well as incompatibilities. > > > Potential Incompatibilities > > * All Corba applications are now moved from the OTP repository > * A new Corba repository will be created https://github.com/erlang > * New applications ftp and tftp, moved from inets > * ssl no longer supports 3_DES cipher suites or RSA-key exchange > cipher suites by default > * erlang:monitor on a primitive node (erl_interface, jinterface, etc) > will no longer fail with badarg exception. Instead a monitor will be > created, but it will only supervise the connection to the node. > > > ?Highlights > > > ?Erts: > > * Enhanced IO scalability > * Support for usage of distribution controller processes for > alternative transports, routing etc > * compact instructions on 64bit systems for code below 4GB 20% less > memory for loaded code > * Rewrite of the efile-driver with NIFs and "Dirty schedulers" > resulting in faster file operations > * non-smp VM removed > * link and monitor optimized for scalability > * os:getenv/putenv now work on thread-safe emulation. No longer in > sync with libc getenv(3). Manual synchronization will be needed. > > > Compiler: > > * Misc compiler optimizations including contributions from the Elixir > team resulting in 10% improvements in benchmarks > * "Tuple calls" have been removed from the run-time system. > * Code such as |f({ok, Val}) -> {ok, Val}| is now automatically > rewritten to |f({ok, Val} = Tuple) -> Tuple.| this reduces code > size, execution time, and removed GC pressure. > * More information in stacktrace from a number of operators > * |erlang:get_stacktrace/0| deprecated to be replaced with |try ... > catch C:R:Stacktrace -> ...| > * Creation of small maps with literal keys optimized. > > > Security: > > * DTLS support in SSL > * Enhanced support for distribution over TLS > * "unsecure" ciphers removed from defaults in SSL and SSH. > * A new option value defined to facilitate implementing exec servers. > Old option kept for compatibility, but now gives errors on stderror. > > > Standard libraries: > > * New API for logging, logger > * New uri_string module for parsing URIs according to "The standard" > * New function lists:search(list,fun/1) -> {ok, Value} | false > * Changed default behaviour of .erlang loading. escript, erlc, > dialyzer and typer no longer load an .erlang at all. > > For more details see > http://erlang.org/download/otp_src_21.0-rc2.readme > > Pre built versions for Windows can be fetched here: > http://erlang.org/download/otp_win32_21.0-rc2.exe > http://erlang.org/download/otp_win64_21.0-rc2.exe > > Online documentation can be browsed here: > http://erlang.org/documentation/doc-10.0-rc2/doc > > The Erlang/OTP source can also be found at GitHub on the official Erlang > repository, > > https://github.com/erlang/otp > > tag: > > OTP-21.0-rc2 > > Thank you for all your contributions! > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From codewiget95@REDACTED Thu May 31 17:24:22 2018 From: codewiget95@REDACTED (Code Wiget) Date: Thu, 31 May 2018 11:24:22 -0400 Subject: [erlang-questions] Eunit testing after starting the application Message-ID: <13db7060-0456-4ec1-94e1-9e5df751498c@Spark> Hey everyone, How do you go about EUnit testing that requires the whole application to be running before the test ? For example, my erlang application starts one pool of processes that connects to a database and another that connects to another server in our backend. To run any real tests, I?m going to need to have one or both of those systems up and running. The way that I?m doing is right now is below, and it is very cluge-y feeling: ok = application:start(compiler), ok = application:start(syntax_tools), ok = application:start(goldrush), ok = application:start(gproc), ok = application:start(gen_logger), ok = application:start(pooler), ok = application:start(quickrand), ok = application:start(re2), ok = application:start(lz4), ok = application:start(uuid), ok = application:start(semver), ok = application:start(snappy), ok = application:start(cqerl), ok = application:start(ecpool), ok = application:start(esockd), ok = application:start(lager_syslog), ok = application:start(lager), This system works because none of the above applications require any environment variables from a config file. So some of my other applications can?t run using this. Is there any way to have eunit run the test after starting the whole application? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathaniel@REDACTED Thu May 31 17:37:12 2018 From: nathaniel@REDACTED (Nathaniel Waisbrot) Date: Thu, 31 May 2018 11:37:12 -0400 Subject: [erlang-questions] Eunit testing after starting the application In-Reply-To: <13db7060-0456-4ec1-94e1-9e5df751498c@Spark> References: <13db7060-0456-4ec1-94e1-9e5df751498c@Spark> Message-ID: <3C155E62-E404-45E6-AF62-0B27C759B7F1@waisbrot.net> I don't have a good answer to your exact question, but I do have two tips: `application:ensure_all_started/1` asks Erlang to figure out how to start all the dependencies in the correct order and handles the case of an application already having been started. And `application:set_env/4` will allow you to programmatically set the Erlang env variables without a config file. > On May 31, 2018, at 11:24 AM, Code Wiget wrote: > > Hey everyone, > > How do you go about EUnit testing that requires the whole application to be running before the test ? > > For example, my erlang application starts one pool of processes that connects to a database and another that connects to another server in our backend. To run any real tests, I?m going to need to have one or both of those systems up and running. The way that I?m doing is right now is below, and it is very cluge-y feeling: > > ok = application:start(compiler), > ok = application:start(syntax_tools), > ok = application:start(goldrush), > ok = application:start(gproc), > ok = application:start(gen_logger), > ok = application:start(pooler), > ok = application:start(quickrand), > ok = application:start(re2), > ok = application:start(lz4), > ok = application:start(uuid), > ok = application:start(semver), > ok = application:start(snappy), > ok = application:start(cqerl), > ok = application:start(ecpool), > ok = application:start(esockd), > ok = application:start(lager_syslog), > ok = application:start(lager), > > > This system works because none of the above applications require any environment variables from a config file. So some of my other applications can?t run using this. Is there any way to have eunit run the test after starting the whole application? > > Thanks! > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.a.vinogradov@REDACTED Thu May 31 17:42:27 2018 From: g.a.vinogradov@REDACTED (Gleb Vinogradov) Date: Thu, 31 May 2018 22:42:27 +0700 Subject: [erlang-questions] Eunit testing after starting the application In-Reply-To: <13db7060-0456-4ec1-94e1-9e5df751498c@Spark> References: <13db7060-0456-4ec1-94e1-9e5df751498c@Spark> Message-ID: You can use CommonTest to bootstrap your application. Set environment variables, start applications etc in CommonTest and run EUnit test then. 2018-05-31 22:24 GMT+07:00 Code Wiget : > Hey everyone, > > How do you go about EUnit testing that requires the whole application to > be running before the test ? > > For example, my erlang application starts one pool of processes that > connects to a database and another that connects to another server in our > backend. To run any real tests, I?m going to need to have one or both of > those systems up and running. The way that I?m doing is right now is below, > and it is very cluge-y feeling: > > ok = application:start(compiler), > ok = application:start(syntax_tools), > ok = application:start(goldrush), > ok = application:start(gproc), > ok = application:start(gen_logger), > ok = application:start(pooler), > ok = application:start(quickrand), > ok = application:start(re2), > ok = application:start(lz4), > ok = application:start(uuid), > ok = application:start(semver), > ok = application:start(snappy), > ok = application:start(cqerl), > ok = application:start(ecpool), > ok = application:start(esockd), > ok = application:start(lager_syslog), > ok = application:start(lager), > > > This system works because none of the above applications require any > environment variables from a config file. So some of my other applications > can?t run using this. Is there any way to have eunit run the test after > starting the whole application? > > Thanks! > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Thu May 31 18:52:25 2018 From: codewiget95@REDACTED (Code Wiget) Date: Thu, 31 May 2018 12:52:25 -0400 Subject: [erlang-questions] Eunit testing after starting the application In-Reply-To: References: <13db7060-0456-4ec1-94e1-9e5df751498c@Spark> Message-ID: Nathaniel, Thank you - ensure_all_started is going to save me dozens of lines of code in other locations =). Gleb, Are you suggesting not doing eunit testing and just running with common tests and using init_per_testcase ? I get that EUnit is easy, but for an application that is already built, when we are trying to test system functionality, maybe it is better to just go full on with the common test suite? On May 31, 2018, 11:42 AM -0400, Gleb Vinogradov , wrote: > You can use CommonTest to bootstrap your application. Set environment variables, start applications etc in CommonTest and run EUnit test then. > > > 2018-05-31 22:24 GMT+07:00 Code Wiget : > > > Hey everyone, > > > > > > How do you go about EUnit testing that requires the whole application to be running before the test ? > > > > > > For example, my erlang application starts one pool of processes that connects to a database and another that connects to another server in our backend. To run any real tests, I?m going to need to have one or both of those systems up and running. The way that I?m doing is right now is below, and it is very cluge-y feeling: > > > > > > ok = application:start(compiler), > > > ok = application:start(syntax_tools), > > > ok = application:start(goldrush), > > > ok = application:start(gproc), > > > ok = application:start(gen_logger), > > > ok = application:start(pooler), > > > ok = application:start(quickrand), > > > ok = application:start(re2), > > > ok = application:start(lz4), > > > ok = application:start(uuid), > > > ok = application:start(semver), > > > ok = application:start(snappy), > > > ok = application:start(cqerl), > > > ok = application:start(ecpool), > > > ok = application:start(esockd), > > > ok = application:start(lager_syslog), > > > ok = application:start(lager), > > > > > > > > > This system works because none of the above applications require any environment variables from a config file. So some of my other applications can?t run using this. Is there any way to have eunit run the test after starting the whole application? > > > > > > Thanks! > > > > > > > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewiget95@REDACTED Thu May 31 19:02:13 2018 From: codewiget95@REDACTED (Code Wiget) Date: Thu, 31 May 2018 13:02:13 -0400 Subject: [erlang-questions] Eunit testing after starting the application In-Reply-To: References: <13db7060-0456-4ec1-94e1-9e5df751498c@Spark> Message-ID: I found this: ERL_AFLAGS="-config etc/server.config" rebar3 eunit This will set the config file to etc/server.config before compiling and running the eunit testing. So in combination with ensure_all_started, then we can start the code necessary to run without the common test framework. This gets around the need to export functions for the common test framework so that we can test the internal functions. On May 31, 2018, 12:52 PM -0400, Code Wiget , wrote: > Nathaniel, > > Thank you - ensure_all_started is going to save me dozens of lines of code in other locations =). > > Gleb, > > Are you suggesting not doing eunit testing and just running with common tests and using init_per_testcase ? I get that EUnit is easy, but for an application that is already built, when we are trying to test system functionality, maybe it is better to just go full on with the common test suite? > > > > On May 31, 2018, 11:42 AM -0400, Gleb Vinogradov , wrote: > > You can use CommonTest to bootstrap your application. Set environment variables, start applications etc in CommonTest and run EUnit test then. > > > > > 2018-05-31 22:24 GMT+07:00 Code Wiget : > > > > Hey everyone, > > > > > > > > How do you go about EUnit testing that requires the whole application to be running before the test ? > > > > > > > > For example, my erlang application starts one pool of processes that connects to a database and another that connects to another server in our backend. To run any real tests, I?m going to need to have one or both of those systems up and running. The way that I?m doing is right now is below, and it is very cluge-y feeling: > > > > > > > > ok = application:start(compiler), > > > > ok = application:start(syntax_tools), > > > > ok = application:start(goldrush), > > > > ok = application:start(gproc), > > > > ok = application:start(gen_logger), > > > > ok = application:start(pooler), > > > > ok = application:start(quickrand), > > > > ok = application:start(re2), > > > > ok = application:start(lz4), > > > > ok = application:start(uuid), > > > > ok = application:start(semver), > > > > ok = application:start(snappy), > > > > ok = application:start(cqerl), > > > > ok = application:start(ecpool), > > > > ok = application:start(esockd), > > > > ok = application:start(lager_syslog), > > > > ok = application:start(lager), > > > > > > > > > > > > This system works because none of the above applications require any environment variables from a config file. So some of my other applications can?t run using this. Is there any way to have eunit run the test after starting the whole application? > > > > > > > > Thanks! > > > > > > > > > > > > > > > > _______________________________________________ > > > > erlang-questions mailing list > > > > erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zelazny@REDACTED Thu May 31 19:31:18 2018 From: zelazny@REDACTED (=?UTF-8?Q?Przemys=c5=82aw_Simon_Zelazny?=) Date: Thu, 31 May 2018 10:31:18 -0700 Subject: [erlang-questions] Eunit testing after starting the application In-Reply-To: References: <13db7060-0456-4ec1-94e1-9e5df751498c@Spark> Message-ID: > > Are you suggesting not doing eunit testing and just running with common > tests and using init_per_testcase ? I get that EUnit is easy, but for an > application that is already built, when we are trying to test system > functionality, maybe it is better to just go full on with the common > test suite? Hi, from my experience, if you've inherited a big, untested application that you want to put tests around, trying to start from a "unit-test" level will be 1) difficult, 2) counterproductive. 1) is because you're trying to capture units of functionality when it's clear that the application was not designed in a modular, isolated fashion. So you end up launching-the-world (or mocking-the-world), which is tedious and leads to.. 2) counterproductive, brittle tests. You're now baking in the coupling that the application has to its dependencies and making that coupling a contract. You're only adding to the snowball that you've got. Instead of the above, I'd recommend putting in place a suite of true end-to-end tests, and capturing the black-box behavior of your system. # Makefile .PHONY: all rel run e2e-test all: rel run e2e-test @echo "E2E tests passed" rel: rebar3 release run: _build/blahblah/yourapp start wait_for_it localhost yourport e2e-test: rebar3 ct Now, your Common Tests will have to assume some things about the system, but these assumptions will only reveal what your clients are assuming anyway. The benefit is you've decoupled knowledge of internals from a behavioral specification of your system. Now, you can go in and re-write/re-factor particular modules, and cover them with true unit tests, eliminating as much coupling as possible. See here: https://mike-bland.com/2011/11/01/small-medium-large.html for a nice overview of testing levels at Google. Cheers, Simon Zelazny > > > > On May 31, 2018, 11:42 AM -0400, Gleb Vinogradov > , wrote: >> You can use CommonTest to bootstrap your application. Set environment >> variables, start applications etc in CommonTest and run EUnit test then. >> >> 2018-05-31 22:24 GMT+07:00 Code Wiget > >: >> >> Hey everyone, >> >> How do you go about EUnit testing that requires the whole >> application to be running before the test ? >> >> For example, my erlang application starts one pool of processes >> that connects to a database and another that connects to another >> server in our backend. To run any real tests, I?m going to need to >> have one or both of those systems up and running. The way that I?m >> doing is right now is below, and it is very cluge-y feeling: >> >> ok = application:start(compiler), >> ok = application:start(syntax_tools), >> ok = application:start(goldrush), >> ok = application:start(gproc), >> ok = application:start(gen_logger), >> ok = application:start(pooler), >> ok = application:start(quickrand), >> ok = application:start(re2), >> ok = application:start(lz4), >> ok = application:start(uuid), >> ok = application:start(semver), >> ok = application:start(snappy), >> ok = application:start(cqerl), >> ok = application:start(ecpool), >> ok = application:start(esockd), >> ok = application:start(lager_syslog), >> ok = application:start(lager), >> >> >> This system works because none of the above applications require >> any environment variables from a config file. So some of my other >> applications can?t run using this. Is there any way to have eunit >> run the test after starting the whole application? >> >> Thanks! >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From max.lapshin@REDACTED Thu May 31 21:05:30 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 31 May 2018 22:05:30 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: It is video, we need to receive packets strictly in order. Yes, I know that UDP can reorder and lose packets, but in normal conditions with hardware headend and plain C linux software on receiver it is possible to achieve almost 100% of deliverability of packets without reordering. We want to achieve something like this with erlang. -------------- next part -------------- An HTML attachment was scrubbed... URL: