From igor.clark@REDACTED Fri Jun 1 01:31:51 2018 From: igor.clark@REDACTED (Igor Clark) Date: Fri, 1 Jun 2018 00:31:51 +0100 Subject: [erlang-questions] How to track down intermittent segfaults in a threaded NIF In-Reply-To: References: <3f0cff19-6f88-87d7-e7be-9b4e6a057ac6@gmail.com> <351EAFAB-D72B-4E54-93F2-43E7A893B20D@elevated-dev.com> <7342393D-9CAA-42C8-AD09-665423BC91D2@elevated-dev.com> <9cdd425a-e8a0-c37e-efe4-932f64417925@gmail.com> <51C26C64-84A8-4B9C-882E-9D52C019172F@elevated-dev.com> <3ac44ee7-1047-164c-08b3-b13fe723806c@gmail.com> Message-ID: <392964c9-37f8-58a5-f837-6898cbba965c@gmail.com> Hey again - just closing the loop here. I had another even less frequent bug that was still triggering erts_exit(ERTS_ABORT_EXIT, ...), only happening once in a blue moon, so I fired up guard malloc with the "+Mea min" option, and this time not only did it give an immediate crash, but it also gave me the exact line number :-)) I can't tell you how useful this is - I've had these intermittent problems only showing up every now and then for a good while, with no real way to track them down, and so they've been grumbling at the back of my mind all that time. So the combination of the erl option and libgmalloc is just an amazing tool for me to hunt down this kind of issue. Thanks so much for telling me about it! Very best, Igor On 30/05/2018 10:19, Igor Clark wrote: > Thanks Dominic - I don't want to count my chickens before they've > hatched, but it looks like guard malloc has pointed me to at least > some bugs even without that VM option. Even though I wasn't getting a > line number in the stack trace, it was already seeming to make the NIF > crash immediately and consistently, so I was able to use a ton of > debug print statements to track down two problems that I hadn't been > able to see before. (One was an enif_alloc() in the wrong place, and > another seems to have been accessing a pointer from a function in a > shared object file, oops.) No way would I have seen them without guard > malloc showing me the way, it's a powerful tool :-) > > So I fixed those two, and right now the app is running as expected > without crashes under guard malloc. I'm pretty sure that I'll come up > against more illegal-access bugs over time, so I'm adding "+Mea min" > to the list of options to use when I find the next one. Thank you. > > Thanks very much also to everyone who replied, particularly Scott for > the guard malloc suggestion & help, and Fred & Tristan for the rebar3 > tips so I could add the necessary CLI options and track down what was > going on. I'm very glad to have been able to ask such experienced > folks for advice, and to have learned about some *extremely* useful > new stuff. > > Cheers, > Igor > > > On 29/05/2018 23:58, Dominic Morneau wrote: >> Can you give it a try with "+Mea min" in erl options? This should >> make Erlang fall back to malloc for all allocators, hopefully making >> guard malloc more effective. >> >> Dominic >> >> 2018?5?30?(?) 5:15 Igor Clark > >: >> >> OK. Thanks very much Scott. I've got all this working using both >> those >> extra options, and it does seem to make the NIF crash a lot >> sooner than >> previously, which is great. But I'm still only seeing >> "process_main" in >> the crashed thread, so I'm not much closer to knowing where the >> illegal >> access is. I wonder if it's in lots of places because of what I'm >> doing >> with the callback and the thread. I hope not. >> >> I'll do some more digging, and tomorrow I'll try out a debug >> emulator >> build as well. >> >> Thanks very much for helping me get this far! >> >> On 29/05/2018 16:31, Scott Ribe wrote: >> >> On May 29, 2018, at 9:16 AM, Igor Clark > > wrote: >> >> >> >> So, do I have this right: the point of the Guard Malloc is to >> make the crash happen at the time of allocation, rather than >> delayed until something trying to access it triggers the >> segfault; so if I get a crash while running like this, I should >> be able to just check in the Console debug log, and the stack >> trace should show where the bug actually is? >> > At the time of the illegal access, not the allocation. Yes, >> that's the point, you get a stack trace showing you illegal access. >> > >> > However, the BEAM allocator will reduce its effectiveness. When >> you malloc in your C code, you get a block set up such that >> accessing just past it (or potentially before it) will cause an >> immediate crash. When you free it, it's then set up such that >> accessing will cause an immediate crash. But if you use Erlang's >> allocation routines, Erlang may malloc a bigger block with those >> protections, then hand out multiple suballocations, and access >> beyond the end of one of those can simply corrupt the next one >> without crashing at that point. >> > >> > You should also be using MallocScribble & MallocPreScribble. >> > >> > >> > >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marthinus.beneke@REDACTED Fri Jun 1 12:45:10 2018 From: marthinus.beneke@REDACTED (Marthinus Beneke) Date: Fri, 1 Jun 2018 12:45:10 +0200 Subject: [erlang-questions] yaws gen_event handler yaws_log_file_h crashed Message-ID: <5f0ce569-2777-8e16-d1ff-966b89497fd1@cellfind.co.za> ? Good day I have an error in yaws that I have been trying to solve to no avail and need some help. I'm using OTP 20 which I installed from source and yaws-2.0.5 that I cloned from github.com/klacke/yaws > Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:1:1] [ds:1:1:10] > [async-threads:10] [hipe] [kernel-poll:false] Everything compiles and the modules start up correctly but the write to the report.log file fails. The file is however created and the .access and .auth files are created and written to. Here is the error > =ERROR REPORT==== 1-Jun-2018::12:09:22 === > ** gen_event handler yaws_log_file_h crashed. > ** Was installed in error_logger > ** Last event was: {info_report,<0.59.0>, > ??????????????????????????????? {<0.33.0>,progress, > ???????????????????????????????? [{application,myapplication}, > ????????????????????????????????? {started_at,myapplication@REDACTED}]}} > ** When handler state == {<0.211.0>,"./report.log",[]} > ** Reason == {function_clause, > ???????????????? [{error_logger_file_h,write_event, > ????????????????????? [{<0.211.0>,"./report.log",[]}, > ?????????????????????? {info_report,<0.59.0>, > ?????????????????????????? {<0.33.0>,progress, > ??????????????????????????? [{application,myapplication}, > ???????????????????????????? {started_at,myapplication@REDACTED}]}}], > ????????????????????? [{file,"error_logger_file_h.erl"},{line,106}]}, > ????????????????? {error_logger_file_h,handle_event,2, > ????????????????????? [{file,"error_logger_file_h.erl"},{line,71}]}, > ????????????????? {gen_event,server_update,4, > ????????????????????? [{file,"gen_event.erl"},{line,574}]}, > ????????????????? {gen_event,server_notify,4, > ????????????????????? [{file,"gen_event.erl"},{line,556}]}, > ????????????????? > {gen_event,handle_msg,6,[{file,"gen_event.erl"},{line,297}]}, > ????????????????? {proc_lib,init_p_do_apply,3, > ????????????????????? [{file,"proc_lib.erl"},{line,247}]}]} Thanks -- Marthinus Beneke Mobile: 083 657 5801 Phone: 011 653 5387 Email: marthinus.beneke@REDACTED www.cellfind.co.za??Email: marthinus.beneke@REDACTED www.cellfind.co.za?? ?? For Disclaimer and Confidentiality note: Click Here Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: RSImage.jpeg Type: image/jpeg Size: 49212 bytes Desc: not available URL: From jchassoul@REDACTED Fri Jun 1 23:28:19 2018 From: jchassoul@REDACTED (Jean Chassoul) Date: Fri, 1 Jun 2018 17:28:19 -0400 Subject: [erlang-questions] RFC The Computer Nonsense Guide Message-ID: Hope you all have fun on Stockholm this week, and now for something completely different: a request for comments and feedback from the Latin American jungle. https://github.com/nonsensews/guide/wiki Jean -------------- next part -------------- An HTML attachment was scrubbed... URL: From bombadil@REDACTED Sun Jun 3 12:14:46 2018 From: bombadil@REDACTED (Manuel A. Rubio) Date: Sun, 03 Jun 2018 12:14:46 +0200 Subject: [erlang-questions] [ANN] Erlang/OTP Volumen II: Las bases de OTP Message-ID: Hello, Sorry for the Spanish content. This is an announcement about a book release in Spanish about Erlang/OTP. This time based mainly in OTP. It's released! You can check: https://erlang-otp.es/volumen-ii/ Kind regards. Manuel Rubio. From alex@REDACTED Sun Jun 3 14:20:58 2018 From: alex@REDACTED (alex@REDACTED) Date: Sun, 3 Jun 2018 08:20:58 -0400 Subject: [erlang-questions] [ANN] Erlang/OTP Volumen II: Las bases de OTP In-Reply-To: References: Message-ID: <57ba3ac3962603d73fafbd3135e2a0a2@smtp.hushmail.com> Awesome job!? Didn't even know about Volume 1, so altogether that's a nice discovery! Thanks, Alex On 06/03/2018 06:14 AM, Manuel A. Rubio wrote: > Hello, > > Sorry for the Spanish content. This is an announcement about a book > release in Spanish about Erlang/OTP. This time based mainly in OTP. > It's released! > > You can check: https://erlang-otp.es/volumen-ii/ > > Kind regards. > Manuel Rubio. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From rafapi@REDACTED Sun Jun 3 14:31:04 2018 From: rafapi@REDACTED (Rafael Pardinas) Date: Sun, 3 Jun 2018 13:31:04 +0100 Subject: [erlang-questions] [ANN] Erlang/OTP Volumen II: Las bases de OTP In-Reply-To: <57ba3ac3962603d73fafbd3135e2a0a2@smtp.hushmail.com> References: <57ba3ac3962603d73fafbd3135e2a0a2@smtp.hushmail.com> Message-ID: Awesome job, indeed! We need more Erlang content in Spanish. Cheers, Rafa On Sun, 3 Jun 2018 at 13:21 alex@REDACTED wrote: > Awesome job! Didn't even know about Volume 1, so altogether that's a > nice discovery! > > > Thanks, > > Alex > > > On 06/03/2018 06:14 AM, Manuel A. Rubio wrote: > > Hello, > > > > Sorry for the Spanish content. This is an announcement about a book > > release in Spanish about Erlang/OTP. This time based mainly in OTP. > > It's released! > > > > You can check: https://erlang-otp.es/volumen-ii/ > > > > Kind regards. > > Manuel Rubio. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Jun 4 17:38:03 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 4 Jun 2018 17:38:03 +0200 Subject: [erlang-questions] Future plans for "async socket accept"? Message-ID: Hello, There is an undocumented method that allows users to accept new connections asynchronously (prim_inet:async_accept then a few extra calls after receiving the socket). This has been used by many people in the past and we're going to look into using it for Ranch/Cowboy. I wonder if it will remain in some form once the network code is moved to use NIFs/dirty schedulers? Thanks, -- Lo?c Hoguin https://ninenines.eu From z@REDACTED Tue Jun 5 14:21:36 2018 From: z@REDACTED (Danil Zagoskin) Date: Tue, 5 Jun 2018 15:21:36 +0300 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code Message-ID: Hi! This is a shout of pain we got while preparing our project for upcoming OTP release. The problem: * We compile our code for production using warnings_as_errors option. This helps us to keep the code tidy. * OTP21 deprecates some functions (erlang:get_stacktrace/0, ssl:ssl_accept/0, may be more) * OTP20 does not support new API (catch C:R:S, ssl:handshake/2) So, to be able to go with both OTP20 and OTP21, we need some hacks. First thought: let's pass {nowarn_deprecated_function, [...]} with Emakefile, letting old code compile smootly in newer OTP. But that cannot be done ? this option is only recognized when given in files. Second thought: OK, let's just add this option to all affected files. But "Warning: erlang:get_stacktrace/0 is not a deprecated function" So, we needed to implement some preprocessor logic which adds nowarn only when compiling with OTP21. Luckily, there is a OTP_RELEASE var defined in OTP21: -ifdef(OTP_RELEASE). -compile({nowarn_deprecated_function, [{erlang, get_stacktrace, 0}]}). -endif. Adding that to tens of files (large project, lots of dependencies in-tree) seemed too ugly, so we implemented a parse_transform which can be added to Emakefile. Parse transform itself: https://gist.github.com/stolen/6a55221ffb906bde712cd939a729718d -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hogberg@REDACTED Tue Jun 5 14:29:52 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Tue, 5 Jun 2018 12:29:52 +0000 Subject: [erlang-questions] Patch package OTP 20.3.7 released Message-ID: <1528201792.13382.88.camel@ericsson.com> Patch Package:???????????OTP 20.3.7 Git Tag:?????????????????OTP-20.3.7 Date:????????????????????2018-06-05 Trouble Report Id:???????OTP-15080, OTP-15085, OTP-15091, OTP-15092 Seq num:???????????????? System:??????????????????OTP Release:?????????????????20 Application:?????????????erl_docgen-0.7.3, erts-9.3.2, inets-6.5.2 Predecessor:?????????????OTP 20.3.6 ?Check out the git tag OTP-20.3.7, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- erl_docgen-0.7.3 ------------------------------------------------ ?--------------------------------------------------------------------- ?The erl_docgen-0.7.3 application can be applied independently of ?other applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15091????Application(s): erl_docgen ???????????????Update makefile so db_funcs.xsl is a part of the ???????????????installed application. ?Full runtime dependencies of erl_docgen-0.7.3: edoc-0.7.13, erts-9.0, ?stdlib-3.4, xmerl-1.3.7 ?--------------------------------------------------------------------- ?--- erts-9.3.2 ------------------------------------------------------ ?--------------------------------------------------------------------- ?The erts-9.3.2 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15080????Application(s): erts ???????????????Fixed bug in enif_binary_to_term which could cause ???????????????memory corruption for immediate terms (atoms, small ???????????????integers, pids, ports, empty lists). ? OTP-15085????Application(s): erts ???????????????Fixed bug in erlang:system_profile/2 that could cause ???????????????superfluous {profile,_,active,_,_} messages for ???????????????terminating processes. ?Full runtime dependencies of erts-9.3.2: kernel-5.0, sasl-3.0.1, ?stdlib-3.0 ?--------------------------------------------------------------------- ?--- inets-6.5.2 ----------------------------------------------------- ?--------------------------------------------------------------------- ?The inets-6.5.2 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15092????Application(s): inets ???????????????inets: httpd - Gracefully handle bad headers ???????????????The option max_headers operated on the individual ???????????????header length instead of the total length of all ???????????????headers. Also headers with empty keys are now ???????????????discarded. ?Full runtime dependencies of inets-6.5.2: erts-6.0, kernel-3.0, ?mnesia-4.12, runtime_tools-1.8.14, ssl-5.3.4, stdlib-2.0 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- From sean.hinde@REDACTED Tue Jun 5 19:51:32 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Tue, 05 Jun 2018 19:51:32 +0200 Subject: [erlang-questions] Binary string literal syntax Message-ID: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> I?ve been writing a lot of Elixir over the last few months (plus Swift, Java, C) and just came back to Erlang. There are a few things I?ve come to very much like about Elixir I think might be quite useful to bring to Erlang. The first, and topic of this email, is arguably trivial, but having to surround <> string literals with <<>> is irritating and not getting any less so. It occurred to me we might usefully steal the sigil notation from Elixir to solve this and add other string literal related syntactic niceties. My proposal would be to add an alternative notation for binary string literals in Erlang along the lines of: ~s?Some binary string? mapping to <<"Some binary string?>> This would open the door to other alternative notations. e.g. Elixir has one for regex friendly strings: ~r/foo|bar/ String literals containing quotes could also benefit from different delimiter characters e.g. ~s'{"Name":"Tom","Age":10}? mapping to <<"{\"Name\":\"Tom\",\"Age\":10}?>> The Elixir docs cover more options: https://elixir-lang.org/getting-started/sigils.html. There are 8 alternative delimiters, and even user defined sigils and interpolation with #{} If we didn?t want to go as far as Elixir style sigils we could just use a single prefix char. LFE uses: #?Some binary string? or maybe ~?Some binary string? If people here on the list think there is anything useful in this, and we can reach some kind of rough consensus on how far it makes sense to go I will put some effort into an EEP. Sean - first post since July 2008. It?s good to be back From zxq9@REDACTED Wed Jun 6 00:45:35 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 06 Jun 2018 07:45:35 +0900 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> Message-ID: <5648284.oqd1qQ7lNO@takoyaki> On 2018?6?5???? 19?51?32? JST Sean Hinde wrote: > I?ve been writing a lot of Elixir over the last few months (plus Swift, Java, C) and just came back to Erlang. There are a few things I?ve come to very much like about Elixir I think might be quite useful to bring to Erlang. > > The first, and topic of this email, is arguably trivial, but having to surround <> string literals with <<>> is irritating and not getting any less so. I like ideas such as this but believe the core language is not the place to implement them. Syntax is like ocean plastic -- it lingers forever, works its way into unexpected places and interacts with the metabolism in surprising ways. Adding syntax to an existing language, especially syntax that represents something that is already one can represent, is *dangerous*. It makes the overall language less useful and each addition dramatically increases the cognitive overhead of learning the language. Typing two different characters two times each, as in <<>> does not exhibit much advantage in keystrokiness over typing a widely gapped sequence ~s''. Magical quote interpretation, as in Python, to avoid the need to escape one or the other kinds of quote literals within a string is handy, but doesn't change much about the difficulty of the language. As Erlang already makes a distinction between single and double quotes, though, diddling with this is a dangerous change (adding syntax) in the interest of glyphy familiarity. Erlang is a small language. A large part of its utility stems from the fact that the core language is TINY and not littered with a ton of conveniences. This should be strongly protected. Compare the universal readability of C vs the wetware-level incompatibilty among various styles of C++ (or Haskell, for that matter); that's astonishing and sad about C++. If Elixir does what you like, then use it. There is a strong chance that Elixir will explore so much in terms of language conveniences that it will blow up the way C++, Ruby, and Perl have. That's fine, because we'll learn a lot and Elixir's grandchildren will really be something! Introducing Elixirisms into Erlang, a stick-in-the-mud, traditional, old, venerable language with a very low learning curve at the level of language complexity, is purely a risk. I would argue that there are *many* places Erlang could benefit from changes, but that inside Erlang is not the place to make them. I'd like to create a "nearly Erlang" language -- 99% "readable as Erlang" but not actually Erlang -- to advance changes like the one you propose. It could compile on its own or simply rewrite to Erlang proper. - Semantic whitespace (no more ant-poop or weirdness with terminators before 'end' or 'after' or 'catch') - Syntactic representation of pipeline checks (or a stdlib pipeline function) to avoid a string of checks forcing newbies into writing highly nested 'case' - Removal of 'if' - A reworking of quote representations to make UTF-8 atoms, strings, binary strings, etc. a bit less noisy (maybe the approach you propose, actually, but I would require a lot of time to think about it to get it right). - A '-pure' declaration (but this could be implemented already in Dialyzer, but just isn't) - etc. The place to do these changes is not in Erlang, but in another language that compiles to Erlang or to BEAM. I'm an Erlang conservationist. Don't corrupt the pristine Erlagnessness with your stays-around-forever-like-marine-plastic syntactic litter! -Craig From jose.valim@REDACTED Wed Jun 6 00:59:58 2018 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Wed, 6 Jun 2018 00:59:58 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <5648284.oqd1qQ7lNO@takoyaki> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <5648284.oqd1qQ7lNO@takoyaki> Message-ID: > There is a strong chance that Elixir will explore so much in terms of language conveniences that it will blow up the way C++, Ruby, and Perl have. Elixir has not added new syntax since v1.0, launched in Sep/2014 (almost 4 years ago, while the language is 6.5 years old). Granted, 4 years is not a long time in terms of programming languages, especially when compared to Erlang/C++/Ruby, but it shows the trend is not pointing to the direction you have hinted. -- *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Jun 6 01:22:07 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 06 Jun 2018 08:22:07 +0900 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <5648284.oqd1qQ7lNO@takoyaki> Message-ID: <10771287.qexmPnxNid@takoyaki> On 2018?6?6???? 0?59?58? JST you wrote: > > There is a strong chance that Elixir will explore so much in terms of > language conveniences that it will blow up the way C++, Ruby, and Perl have. > > Elixir has not added new syntax since v1.0, launched in Sep/2014 (almost 4 > years ago, while the language is 6.5 years old). > > Granted, 4 years is not a long time in terms of programming languages, > especially when compared to Erlang/C++/Ruby, but it shows the trend is not > pointing to the direction you have hinted. Don't take this the wrong way, because I think what you've done is great overall: I mean v1.0 itself represents a syntactic explosion (but it couldn't have happened any other way). You were riding the wave of community language organic evolution, and some weirdness always grew out of that. It is impossible for that *not* to happen. Even the Scheme definition has weird corners! If you could write Elixir 2.0 from scratch, regardless what user expectations you might crush, would it come out *exactly* the same way? I doubt it. You've learned a HUGE amount about language design through this process and surely you would hope to implement some of that and clean up the corners eventually. 4 years isn't long and I would be absolutely shocked if there are *no* syntactic changes to v1.x later on down the road, doubly shocked if there is never an Elixir v2, and triply shocked if a few linguistic earthquakes don't happen if or when you decide to abdicate as Elixir's benevolent dictator. (Please never abdicate, by the way. That's a scary thought.) This isn't an indictment of Elixir -- I don't mean it that way -- language design is HARD. People just can't believe how incredibly hard it is to design a sane language. That's why I am extremely conservative with language syntax suggestions once a language is already in broad use. There are strong, princpled reasons you *haven't* added syntax to Elixir since v1.0, right? That's my point. -Craig From soverdor@REDACTED Wed Jun 6 01:32:27 2018 From: soverdor@REDACTED (Sam Overdorf) Date: Tue, 5 Jun 2018 16:32:27 -0700 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <10771287.qexmPnxNid@takoyaki> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <5648284.oqd1qQ7lNO@takoyaki> <10771287.qexmPnxNid@takoyaki> Message-ID: Has anyone ever considered making a string a type and not a list? Thanks, Sam Overdorf On Tue, Jun 5, 2018 at 4:22 PM, wrote: > On 2018?6?6???? 0?59?58? JST you wrote: >> > There is a strong chance that Elixir will explore so much in terms of >> language conveniences that it will blow up the way C++, Ruby, and Perl have. >> >> Elixir has not added new syntax since v1.0, launched in Sep/2014 (almost 4 >> years ago, while the language is 6.5 years old). >> >> Granted, 4 years is not a long time in terms of programming languages, >> especially when compared to Erlang/C++/Ruby, but it shows the trend is not >> pointing to the direction you have hinted. > > Don't take this the wrong way, because I think what you've done is great overall: I mean v1.0 itself represents a syntactic explosion (but it couldn't have happened any other way). You were riding the wave of community language organic evolution, and some weirdness always grew out of that. It is impossible for that *not* to happen. Even the Scheme definition has weird corners! > > If you could write Elixir 2.0 from scratch, regardless what user expectations you might crush, would it come out *exactly* the same way? I doubt it. You've learned a HUGE amount about language design through this process and surely you would hope to implement some of that and clean up the corners eventually. > > 4 years isn't long and I would be absolutely shocked if there are *no* syntactic changes to v1.x later on down the road, doubly shocked if there is never an Elixir v2, and triply shocked if a few linguistic earthquakes don't happen if or when you decide to abdicate as Elixir's benevolent dictator. > > (Please never abdicate, by the way. That's a scary thought.) > > This isn't an indictment of Elixir -- I don't mean it that way -- language design is HARD. People just can't believe how incredibly hard it is to design a sane language. That's why I am extremely conservative with language syntax suggestions once a language is already in broad use. There are strong, princpled reasons you *haven't* added syntax to Elixir since v1.0, right? That's my point. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jose.valim@REDACTED Wed Jun 6 08:37:07 2018 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Wed, 6 Jun 2018 08:37:07 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <10771287.qexmPnxNid@takoyaki> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <5648284.oqd1qQ7lNO@takoyaki> <10771287.qexmPnxNid@takoyaki> Message-ID: Right, all languages have syntax weirdness. And yes, Elixir has more syntactical constructs than Erlang, even being much younger. But there is still a very long way to become comparable to languages like C++ and Perl. And if Erlang can avoid further syntax growth through discipline, I don?t see any reason why Elixir can?t either. :) Elixir was designed with a macro system and AST in mind, exactly so we can have a core of constructs and derive everything else from this core. That?s the reason why we haven?t added further syntax, because it was designed so we don?t have to. So I would be surprised if Elixir v2.0 includes more than a couple new constructs, if any. The most likely chance of getting new constructs is because Erlang got them too (say we got a new data type, like when we got maps). I am fully aware that 4 years is very little time to extract trends from and guess about the future, but I am considerably more optimistic about how this particular aspect of Elixir will grow over time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hinde@REDACTED Wed Jun 6 10:01:38 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Wed, 06 Jun 2018 10:01:38 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <5648284.oqd1qQ7lNO@takoyaki> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <5648284.oqd1qQ7lNO@takoyaki> Message-ID: <78904A5A-7E6D-4E1F-8105-3B10B4F7DEEB@mac.com> > On 6 Jun 2018, at 00:45, zxq9@REDACTED wrote: > > On 2018?6?5???? 19?51?32? JST Sean Hinde wrote: >> I?ve been writing a lot of Elixir over the last few months (plus Swift, Java, C) and just came back to Erlang. There are a few things I?ve come to very much like about Elixir I think might be quite useful to bring to Erlang. >> >> The first, and topic of this email, is arguably trivial, but having to surround <> string literals with <<>> is irritating and not getting any less so. > > I like ideas such as this but believe the core language is not the place to implement them. > > Syntax is like ocean plastic -- it lingers forever, works its way into unexpected places and interacts with the metabolism in surprising ways. Adding syntax to an existing language, especially syntax that represents something that is already one can represent, is *dangerous*. It makes the overall language less useful and each addition dramatically increases the cognitive overhead of learning the language. I have happy memories of waiting eagerly each year to see what new things would be added to the language. The bit syntax was a truly stunning addition. When sometime later the ability to write literal strings inside a binary was added it was pretty convenient. The Erlang team has not shied away from adding features where they push the language forward in its practicality for dealing with the protocol related problems of the day. Back then it was bit twiddling, these days protocols are more string based. > > Typing two different characters two times each, as in <<>> does not exhibit much advantage in keystrokiness over typing a widely gapped sequence ~s''. Magical quote interpretation, as in Python, to avoid the need to escape one or the other kinds of quote literals within a string is handy, but doesn't change much about the difficulty of the language. As Erlang already makes a distinction between single and double quotes, though, diddling with this is a dangerous change (adding syntax) in the interest of glyphy familiarity. Keystroke saving was not really a motivation, though having to add them at both ends of a string is a bunch of navigation as well as the extra keystrokes. > > Erlang is a small language. A large part of its utility stems from the fact that the core language is TINY and not littered with a ton of conveniences. This should be strongly protected. Compare the universal readability of C vs the wetware-level incompatibilty among various styles of C++ (or Haskell, for that matter); that's astonishing and sad about C++. > > If Elixir does what you like, then use it. There is a strong chance that Elixir will explore so much in terms of language conveniences that it will blow up the way C++, Ruby, and Perl have. That's fine, because we'll learn a lot and Elixir's grandchildren will really be something! Introducing Elixirisms into Erlang, a stick-in-the-mud, traditional, old, venerable language with a very low learning curve at the level of language complexity, is purely a risk. I guess there is some risk that adding a nicer way to write binary strings would break the camel?s back and make the language more difficult to learn. I happen to believe that this change would have the opposite affect. It ought to reduce the cognitive overhead for newcomers to the language (?Why do I have to write these weird angle brackets just to get a ?normal? string??), and make the choice between the two string forms less biased by syntax and more based on utility for the rest of us. I can see your point for the full scope of Elixir sigils, but I think there is a level of addition that would simply just help. Maybe just the #?Syntax" > > I'm an Erlang conservationist. Don't corrupt the pristine Erlagnessness with your stays-around-forever-like-marine-plastic syntactic litter! lol. I also love the considered pace of language additions. Maps took a pretty long time to finalise and they have turned out nice. Having written a decent amount of C++ in the last 10 years I am the last person who would want to wish that on the language :) > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sean.hinde@REDACTED Wed Jun 6 11:41:01 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Wed, 06 Jun 2018 11:41:01 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> Message-ID: <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> I had an off list mail with some questions that I?d like to share my answers to The first question was about my use of the word <> Modern had a few thoughts behind it: - It depends on the greyness of your beard ;) - The great re-write of libraries dealing with external string data as input to use binaries instead of strings (especially the json libs) - That UTF-8 has emerged as the universal standard for string data It means that dealing with modern Erlang libs and modern data drives the use of binary strings as the default way of writing strings in Erlang programs where we would once have just used list of int strings > > Anyway, I feel like Erlang is even less of a string-wrangling scripting language than Elixir and so I think I'd find sigils even less helpful. As a protocol wrangling language I would argue Erlang has no peers, but many more protocols are string based now than when the bit syntax was invented. > > >> LFE uses: >> >> #?Some binary string? > > > That one's pretty attractive to me. I could see myself writing them that way. One vote in that direction. Thank you ! > My question, though, is what are the trade-offs of using binaries vs list-of-ints for strings? I think the syntax ought to push people towards the "best" answer. Erlang seems to say list-of-ints and Elixir definitely says binaries. I think if we had a nicer way to write strings as binaries it would push the needle pretty far in the direction of that as the best answer in a lot more cases. I don?t see a lot of reasons not to use binary strings as the preferred option - they take less memory, they can be compared, indexed, searched etc much more efficiently than lists as strings Sean From z@REDACTED Wed Jun 6 12:16:53 2018 From: z@REDACTED (Danil Zagoskin) Date: Wed, 6 Jun 2018 13:16:53 +0300 Subject: [erlang-questions] compile options+documentation consistency Message-ID: Hi! I had a simple task: get an AST of a module (.erl) after preprocessing it (includes + parse_transforms). I read the docs: http://erlang.org/doc/man/compile.html This is how I first understood some options: * 'P' ? get a transformed code instead of a compiled module * 'binary' ? get a result in return value instead of a file So, I run compile:file("example.erl", ['P', binary]) and indeed get an AST in third position of returned tuple. Surprise: example.P file is still written on disk, and it contains Erlang code, not the returned AST. Reading the compile.erl I found an undocumented 'to_pp' option. Passing 'to_pp' alone also generates example.P file, but this time it contains AST. Passing both 'to_pp' and 'binary' works as I wanted ? AST in returned tuple, no files written. My suggestions: * make 'binary' (well, naming legacy) always switch from only writing to file to only returning result. * associate every content with its own file extension (e.g. example.ast for AST) * document 'to_pp' and other options This would make using compiler in non-trivial way easier: * choose at which pass should compiler stop * choose where the result goes Any thoughts on this? -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Wed Jun 6 13:00:03 2018 From: z@REDACTED (Danil Zagoskin) Date: Wed, 6 Jun 2018 14:00:03 +0300 Subject: [erlang-questions] compile options+documentation consistency In-Reply-To: References: Message-ID: One thing more (to not create extra thread) In the documentations there are notes describing some options behaviour, e.g. > The options {nowarn_unused_function, FAs}, {nowarn_bif_clash, FAs}, and {nowarn_deprecated_function, MFAs} > are only recognized when given in files. They are not affected by options > warn_unused_function, warn_bif_clash, or warn_deprecated_function. It's hard to notice them if you read about affected options ? this note is almost 2 screens below the nowarn_deprecated_function description. Maybe it would be easier to read the note if affected options had a link to it, e.g. > {nowarn_deprecated_function, MFAs} > > (This option works only when given in a file and has a precedence over warn_deprecated_function option. See Note [4] for details) > Turns off warnings for calls to deprecated functions likenowarn_deprecated_function > does, but only for the mentioned functions. MFAs is a tuple {Module,Name,Arity} or a list of such tuples. On Wed, Jun 6, 2018 at 1:16 PM, Danil Zagoskin wrote: > Hi! > > I had a simple task: get an AST of a module (.erl) after preprocessing it > (includes + parse_transforms). > > I read the docs: http://erlang.org/doc/man/compile.html > This is how I first understood some options: > * 'P' ? get a transformed code instead of a compiled module > * 'binary' ? get a result in return value instead of a file > > So, I run compile:file("example.erl", ['P', binary]) and indeed get an AST > in third position of returned tuple. > Surprise: example.P file is still written on disk, and it contains Erlang > code, not the returned AST. > > Reading the compile.erl I found an undocumented 'to_pp' option. > Passing 'to_pp' alone also generates example.P file, but this time it > contains AST. > Passing both 'to_pp' and 'binary' works as I wanted ? AST in returned > tuple, no files written. > > My suggestions: > * make 'binary' (well, naming legacy) always switch from only writing to > file to only returning result. > * associate every content with its own file extension (e.g. example.ast > for AST) > * document 'to_pp' and other options > > This would make using compiler in non-trivial way easier: > * choose at which pass should compiler stop > * choose where the result goes > > > Any thoughts on this? > > -- > Danil Zagoskin | z@REDACTED > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Jun 6 13:01:54 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 6 Jun 2018 13:01:54 +0200 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: References: Message-ID: For a larger project, I stumbled into the following while trying to build it on OTP-21.0-rc2-69-g9ae2044073: * `erlexec` sets `warnings_as_errors` which fails due to `erlang:get_stacktrace/0` * `meck` sets `warnings_as_errors` which fails due to `erlang:get_stacktrace/0` * `ranch_proxy_protocol` sets `warnings as errors which fails due to `ssl:ssl_accept/3` * `jose` sets `warnings_as_errors` which fails due to `erlang:get_stacktrace/0` * `cowboy_cors` sets `warnings_as_errors` which fails due to `erlang:get_stacktrace/0` In effect, the warnings_as_errors is now a useless option on any system which wants an upgrade-path from 20 to 21. If you start using the new syntax straight away, you lock out users so they can't use 20 anymore, and in any software project, some system has to support multiple versions in order to do code upgrade[0]. Current fix for the above is to use rebar3's overrides: {overrides, [{override, erlexec, [{erl_opts, [debug_info]}]}, {override, ranch_proxy_protocol, [{erl_opts, [debug_info]}]}, {override, cowboy_cors, [{erl_opts, [debug_info]}]}, {override, jose, [{erl_opts, [debug_info]}]}, {override, meck, [{erl_opts, [debug_info]}]}] }. which plugs the hole for now. But I'm a bit hesitant to just write those projects since I'm not sure I have a good, easily implemented transition path here. I think one might be able to use a bit of macro-magic to accept the particular deprecation warning on 21 (only!) as not-an-error-and-intended-for-now, so one can postpone the 21 upgrade a bit in the software. NOTE: This isn't a problem in fully vendored software where you control everything. You can simply define when you upgrade and handle all dependencies directly. But in a setting where other people rely on your libraries, it is usually a bit more flexibile to allow them the ability to run on the current version as well as the version one level back. [0] Note this is somewhat similar to a code_change/3: Some function needs to understand how to transform old data to new data and thus work with both versions for a while. On Tue, Jun 5, 2018 at 2:22 PM Danil Zagoskin wrote: > Hi! > > This is a shout of pain we got while preparing our project for upcoming > OTP release. > > The problem: > * We compile our code for production using warnings_as_errors option. > This helps us to keep the code tidy. > * OTP21 deprecates some functions (erlang:get_stacktrace/0, > ssl:ssl_accept/0, may be more) > * OTP20 does not support new API (catch C:R:S, ssl:handshake/2) > > So, to be able to go with both OTP20 and OTP21, we need some hacks. > > First thought: let's pass {nowarn_deprecated_function, [...]} with > Emakefile, letting old code > compile smootly in newer OTP. > But that cannot be done ? this option is only recognized when given in > files. > > Second thought: OK, let's just add this option to all affected files. > But "Warning: erlang:get_stacktrace/0 is not a deprecated function" > > So, we needed to implement some preprocessor logic which adds nowarn only > when compiling with OTP21. > Luckily, there is a OTP_RELEASE var defined in OTP21: > -ifdef(OTP_RELEASE). > -compile({nowarn_deprecated_function, [{erlang, get_stacktrace, 0}]}). > -endif. > > Adding that to tens of files (large project, lots of dependencies in-tree) > seemed too ugly, > so we implemented a parse_transform which can be added to Emakefile. > > Parse transform itself: > https://gist.github.com/stolen/6a55221ffb906bde712cd939a729718d > > -- > Danil Zagoskin | z@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Jun 6 13:12:25 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 6 Jun 2018 13:12:25 +0200 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: References: Message-ID: It's worth noting that you did not have an issue with Cowboy, Ranch and friends. That's because the default policy in Erlang.mk is to NOT use warnings_as_errors for dependencies (and as a result this option is not included in the generated rebar.config). This saves a lot of headaches. erlang:get_stacktrace/0 is notable in how widely used it is but issues with warnings_as_error have always existed for as long as I've been doing Erlang. Tools should really have a saner default policy for this. The build shouldn't break for your users just because a new warning was introduced. On 06/06/2018 01:01 PM, Jesper Louis Andersen wrote: > For a larger project, I stumbled into the following while trying to > build it on OTP-21.0-rc2-69-g9ae2044073: > > * `erlexec` sets `warnings_as_errors` which fails due to > `erlang:get_stacktrace/0` > * `meck` sets `warnings_as_errors` which fails due to > `erlang:get_stacktrace/0` > * `ranch_proxy_protocol` sets `warnings as errors which fails due to > `ssl:ssl_accept/3` > * `jose` sets `warnings_as_errors` which fails due to > `erlang:get_stacktrace/0` > * `cowboy_cors` sets `warnings_as_errors` which fails due to > `erlang:get_stacktrace/0` > > In effect, the warnings_as_errors is now a useless option on any system > which wants an upgrade-path from 20 to 21. If you start using the new > syntax straight away, you lock out users so they can't use 20 anymore, > and in any software project, some system has to support multiple > versions in order to do code upgrade[0]. > > Current fix for the above is to use rebar3's overrides: > > {overrides, > ??? [{override, erlexec, [{erl_opts, [debug_info]}]}, > ???? {override, ranch_proxy_protocol, [{erl_opts, [debug_info]}]}, > ???? {override, cowboy_cors, [{erl_opts, [debug_info]}]}, > ???? {override, jose, [{erl_opts, [debug_info]}]}, > ???? {override, meck, [{erl_opts, [debug_info]}]}] > }. > > which plugs the hole for now. But I'm a bit hesitant to just write those > projects since I'm not sure I have a good, easily implemented transition > path here. I think one might be able to use a bit of macro-magic to > accept the particular deprecation warning on 21 (only!) as > not-an-error-and-intended-for-now, so one can postpone the 21 upgrade a > bit in the software. > > NOTE: This isn't a problem in fully vendored software where you control > everything. You can simply define when you upgrade and handle all > dependencies directly. But in a setting where other people rely on your > libraries, it is usually a bit more flexibile to allow them the ability > to run on the current version as well as the version one level back. > > [0] Note this is somewhat similar to a code_change/3: Some function > needs to understand how to transform old data to new data and thus work > with both versions for a while. > > On Tue, Jun 5, 2018 at 2:22 PM Danil Zagoskin > wrote: > > Hi! > > This is a shout of pain we got while preparing our project for > upcoming OTP release. > > The problem: > ?* We compile our code for production using?warnings_as_errors > option. This helps us to keep the code tidy. > ?* OTP21 deprecates some functions (erlang:get_stacktrace/0, > ssl:ssl_accept/0, may be more) > ?* OTP20 does not support new API (catch C:R:S, ssl:handshake/2) > > So, to be able to go with both OTP20 and OTP21, we need some hacks. > > First thought: let's pass {nowarn_deprecated_function, [...]} with > Emakefile, letting old code > compile smootly in newer OTP. > But that cannot be done ? this option is?only recognized when given > in files. > > Second thought: OK, let's just add this option to all affected files. > But "Warning: erlang:get_stacktrace/0 is not a deprecated function" > > So, we needed to implement some preprocessor logic which adds nowarn > only when compiling with OTP21. > Luckily, there is a?OTP_RELEASE var defined in OTP21: > -ifdef(OTP_RELEASE). > -compile({nowarn_deprecated_function, [{erlang, get_stacktrace, 0}]}). > -endif. > > Adding that to tens of files (large project, lots of dependencies > in-tree) seemed too ugly, > so we implemented a parse_transform which can be added to Emakefile. > > Parse transform itself: > https://gist.github.com/stolen/6a55221ffb906bde712cd939a729718d > > -- > Danil Zagoskin | z@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From mononcqc@REDACTED Wed Jun 6 13:18:58 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 6 Jun 2018 07:18:58 -0400 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: References: Message-ID: <20180606111857.GA1083@ferdmbp.local> On 06/06, Lo?c Hoguin wrote: >It's worth noting that you did not have an issue with Cowboy, Ranch >and friends. That's because the default policy in Erlang.mk is to NOT >use warnings_as_errors for dependencies (and as a result this option >is not included in the generated rebar.config). > >This saves a lot of headaches. erlang:get_stacktrace/0 is notable in >how widely used it is but issues with warnings_as_error have always >existed for as long as I've been doing Erlang. Tools should really >have a saner default policy for this. The build shouldn't break for >your users just because a new warning was introduced. > Rebar3 leaves the warnings that libs put for themselves there. There's a PR currently going about adding a 'del' override, which would let someone remove the option by specifying: {overrides, [ {del, [ {erl_opts, [warning_as_errors]} ]} ]} and would let you instantly remove the option from all dependencies rather than having to go through the override pattern Jesper has shown here. If/Once merged, it would help address the problem. I'm a bit hesitant to go and disable some options _by default_ when library authors have set them there for a reason. Now for the case of get_stacktrace(), macros are pretty much the only way we've found to deal with this. I'm a bit disappointed that the issue where the change was made had this debated a long time (https://github.com/erlang/otp/pull/1783) but the changes to the language have been railroaded without regards to community complaints there. Rebar3 ended up going with a macro-heavy approach suggested by @okeuday in here: https://github.com/erlang/rebar3/pull/1773/files Libraries like GPB had to resort to more trickery since the files can be static and embedded in other projects without pre-defined macros: https://github.com/tomas-abrahamsson/gpb/issues/134 This ended up being a big pain in the ass for a lot of people due to the very aggressive deprecation schedule. From jesper.louis.andersen@REDACTED Wed Jun 6 13:23:57 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 6 Jun 2018 13:23:57 +0200 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: References: Message-ID: On Wed, Jun 6, 2018 at 1:12 PM Lo?c Hoguin wrote: > It's worth noting that you did not have an issue with Cowboy, Ranch and > friends. That's because the default policy in Erlang.mk is to NOT use > warnings_as_errors for dependencies (and as a result this option is not > included in the generated rebar.config). > > IIRC, Rebar doesn't include the option by default either. But people like to add that option as a way to rid themselves of warnings in their projects. Normally, this is a fine and sensible thing to do, but I'm currently a bit torn on its usefulness in libraries other people rely on, and I might want to recommend people not adding it to those. Generally I think we agree here. Individual projects can override this setting when you assemble the software on a grander scale and force warnings_as_errors. It is always easier to add strictness at a higher level. But even then: if you don't control all of the source code, you must be prepared for parts of it lagging behind in warning count and/or quality. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathaniel@REDACTED Wed Jun 6 13:36:54 2018 From: nathaniel@REDACTED (Nathaniel Waisbrot) Date: Wed, 6 Jun 2018 07:36:54 -0400 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: References: Message-ID: <3F1DC711-C223-4146-8416-175FCF71632A@waisbrot.net> > IIRC, Rebar doesn't include the option by default either. But people like to add that option as a way to rid themselves of warnings in their projects. Normally, this is a fine and sensible thing to do, but I'm currently a bit torn on its usefulness in libraries other people rely on, and I might want to recommend people not adding it to those. If a project has tests (especially if they're run automatically on code changes), rebar3 can use profiles: {profiles, [ {test, [ {erl_opts, [warnings_as_errors]} ]} ]}. so that way your CI tests should fail and keep your project clean from errors. But at the same time it won't break as a dependency. From nathaniel@REDACTED Wed Jun 6 13:37:23 2018 From: nathaniel@REDACTED (Nathaniel Waisbrot) Date: Wed, 6 Jun 2018 07:37:23 -0400 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> Message-ID: Thanks for copying my email to the list, Sean. That's what happens when I stay up late writing emails. > - That UTF-8 has emerged as the universal standard for string data I think this is an important point. Your bin-string-marker proposal would actually be equivalent, I think, to <<"some string"/utf8>> which is a little more of a mouthful and therefore a better argument in favor of it. From essen@REDACTED Wed Jun 6 13:38:36 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 6 Jun 2018 13:38:36 +0200 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: <20180606111857.GA1083@ferdmbp.local> References: <20180606111857.GA1083@ferdmbp.local> Message-ID: <06d65fa4-3b1c-35c3-6d04-63652ab7edd3@ninenines.eu> On 06/06/2018 01:18 PM, Fred Hebert wrote: > On 06/06, Lo?c Hoguin wrote: >> It's worth noting that you did not have an issue with Cowboy, Ranch >> and friends. That's because the default policy in Erlang.mk is to NOT >> use warnings_as_errors for dependencies (and as a result this option >> is not included in the generated rebar.config). >> >> This saves a lot of headaches. erlang:get_stacktrace/0 is notable in >> how widely used it is but issues with warnings_as_error have always >> existed for as long as I've been doing Erlang. Tools should really >> have a saner default policy for this. The build shouldn't break for >> your users just because a new warning was introduced. >> > > Rebar3 leaves the warnings that libs put for themselves there. > > There's a PR currently going about adding a 'del' override, which would > let someone remove the option by specifying: > > {overrides, [ > ?? {del, [ > ?????? {erl_opts, [warning_as_errors]} > ?? ]} > ]} > > and would let you instantly remove the option from all dependencies > rather than having to go through the override pattern Jesper has shown > here. > > If/Once merged, it would help address the problem. I'm a bit hesitant to > go and disable some options _by default_ when library authors have set > them there for a reason. They have a reason of course: this option is great during development. Everyone should use it. Building as a dependency is a different context though and having your dependencies break every time there's a new OTP version is not the best user experience. I suppose Rebar3 has a way to have stricter compilation profiles for use during development, but I haven't seen much use of that. So it falls down to each dependency's user to fix those issues, and that's a problem. It being a default may or may not be a good idea, I don't know. All I do know is that nobody complained and nobody asked for an option to disable that behavior so I think it's good enough for now. It's one less problem to worry about. Rebar has more users and with a different mindset so the same may not apply. -- Lo?c Hoguin https://ninenines.eu From vladdu55@REDACTED Wed Jun 6 14:00:20 2018 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 6 Jun 2018 14:00:20 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> Message-ID: Hi! I have a few thoughts about this. I would favor the proposed syntax, but not if things don't get simpler. What I mean is that there's more to consider. - Some modules don't handle binary strings, but lists of chars; most notably erl_scan. If the syntaxes are too close, it might be even more confusing when to use which form. - The new string functions work with strings as sequences of lexemes. The "list strings" are lists of characters, so for example calling length() on the two representations of the same text may not return the same value. Most notably, CRLF is a lexeme, but two characters. - When working with a textual protocol, it's still quite often that one would use <<"prefix"/utf8, Rest/binary>>, where the current syntax still has to be used. It might be confusing? - The predefined type string() is still [char()], and for binary strings there is unicode:chardata(), which in not necessarily obvious (as these are handled by the string module). best regards, Vlad On Wed, Jun 6, 2018 at 1:37 PM Nathaniel Waisbrot wrote: > Thanks for copying my email to the list, Sean. That's what happens when I > stay up late writing emails. > > > > - That UTF-8 has emerged as the universal standard for string data > > > I think this is an important point. Your bin-string-marker proposal would > actually be equivalent, I think, to <<"some string"/utf8>> which is a > little more of a mouthful and therefore a better argument in favor of it. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Wed Jun 6 14:21:30 2018 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Wed, 6 Jun 2018 14:21:30 +0200 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: <06d65fa4-3b1c-35c3-6d04-63652ab7edd3@ninenines.eu> References: <20180606111857.GA1083@ferdmbp.local> <06d65fa4-3b1c-35c3-6d04-63652ab7edd3@ninenines.eu> Message-ID: FWIW, Mix also disables warnings as errors for both Erlang (Rebar) and Elixir (Mix) dependencies, for exactly the same reasons outlined by Lo?c. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Wed, Jun 6, 2018 at 1:38 PM, Lo?c Hoguin wrote: > On 06/06/2018 01:18 PM, Fred Hebert wrote: > >> On 06/06, Lo?c Hoguin wrote: >> >>> It's worth noting that you did not have an issue with Cowboy, Ranch and >>> friends. That's because the default policy in Erlang.mk is to NOT use >>> warnings_as_errors for dependencies (and as a result this option is not >>> included in the generated rebar.config). >>> >>> This saves a lot of headaches. erlang:get_stacktrace/0 is notable in how >>> widely used it is but issues with warnings_as_error have always existed for >>> as long as I've been doing Erlang. Tools should really have a saner default >>> policy for this. The build shouldn't break for your users just because a >>> new warning was introduced. >>> >>> >> Rebar3 leaves the warnings that libs put for themselves there. >> >> There's a PR currently going about adding a 'del' override, which would >> let someone remove the option by specifying: >> >> {overrides, [ >> {del, [ >> {erl_opts, [warning_as_errors]} >> ]} >> ]} >> >> and would let you instantly remove the option from all dependencies >> rather than having to go through the override pattern Jesper has shown here. >> >> If/Once merged, it would help address the problem. I'm a bit hesitant to >> go and disable some options _by default_ when library authors have set them >> there for a reason. >> > > They have a reason of course: this option is great during development. > Everyone should use it. > > Building as a dependency is a different context though and having your > dependencies break every time there's a new OTP version is not the best > user experience. I suppose Rebar3 has a way to have stricter compilation > profiles for use during development, but I haven't seen much use of that. > So it falls down to each dependency's user to fix those issues, and that's > a problem. > > It being a default may or may not be a good idea, I don't know. All I do > know is that nobody complained and nobody asked for an option to disable > that behavior so I think it's good enough for now. It's one less problem to > worry about. Rebar has more users and with a different mindset so the same > may not apply. > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hinde@REDACTED Wed Jun 6 14:56:05 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Wed, 06 Jun 2018 14:56:05 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> Message-ID: <7609E2E2-7FAC-476F-A81C-1C894FA02A60@mac.com> > On 6 Jun 2018, at 14:00, Vlad Dumitrescu wrote: > > Hi! > > I have a few thoughts about this. I would favor the proposed syntax, but not if things don't get simpler. What I mean is that there's more to consider. I was aware of having missed a few details, and aware I was undoubtedly unaware of more :) > > - Some modules don't handle binary strings, but lists of chars; most notably erl_scan. If the syntaxes are too close, it might be even more confusing when to use which form. Very true, though equally some modules don?t handle lists of chars. erl_scan is a big one, but I guess we are all used to the endless round of list to binary and vice versa in these cases. > - The new string functions work with strings as sequences of lexemes. The "list strings" are lists of characters, so for example calling length() on the two representations of the same text may not return the same value. Most notably, CRLF is a lexeme, but two characters. That is a big question. How should they be represented? I was happily assuming UTF-8, but maybe it would make more sense for them to be compatible with the new string module and be stored as lexeme sequences. Looking around it seems there are a good range of sensible options. Elixir defaults to string literals being utf8, Swift uses Unicode scalars in their internal string representation forcing conversion to get a byte based representation. With my protocol hat on I think I would pick utf8 as that is the most likely external representation and in many cases we would never need to convert and hence be efficient, but I can see arguments for this being poor design for a language. > - When working with a textual protocol, it's still quite often that one would use <<"prefix"/utf8, Rest/binary>>, where the current syntax still has to be used. It might be confusing? <<#?prefix?, Rest/binary>> ? Definitely room for deeper thought here. > - The predefined type string() is still [char()], and for binary strings there is unicode:chardata(), which in not necessarily obvious (as these are handled by the string module). There is a type for unicode_binary() in the unicode module which refers to a utf8 binary string. The unicode.erl docs go as far as saying: "The default Unicode encoding in Erlang is in binaries UTF-8, which is also the format in which built-in functions and libraries in OTP expect to find binary Unicode data? There is also a strange example in the string.erl document where this binary <<"abc..????>> is not stored as UTF-8 but instead as latin-1. Having an unambiguous way to represent a UTF-8 string literal would also clear this up. That seems to point in a clear direction. Excellent input, thank you. Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Wed Jun 6 15:29:35 2018 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 6 Jun 2018 15:29:35 +0200 Subject: [erlang-questions] Build older versions with right openssl Message-ID: Hi all, I have a need to rebuild older versions of Erlang on Ubuntu and it works fine for v20, but for the others I get "no usable openssl found". I assume that's because a specific version of the libs is required (what I have is 1.1.0). I need the crypto application available. How can I make it work? I have little experience in this area... best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Wed Jun 6 15:39:57 2018 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 6 Jun 2018 16:39:57 +0300 Subject: [erlang-questions] Build older versions with right openssl In-Reply-To: References: Message-ID: Hi, This is how I build R12B5 on CentOS7 in Docker. ENV ERLANG_VERSION R12B-5 ENV openssl openssl-1.0.2o COPY src/${openssl}.tar.gz /tmp/ RUN cat /tmp/${openssl}.tar.gz | tar xfz - && cd ${openssl} && \ ??? ./config --prefix=/usr -fpic shared && \ ??? make && \ ??? make install && \ ??? ldconfig && \ ??? cd .. && rm -rf ${openssl} /tmp/${openssl}.tar.gz RUN curl -S -L http://www.erlang.org/download/otp_src_${ERLANG_VERSION}.tar.gz | tar xz && \ ??? cd otp_src_${ERLANG_VERSION}/ && \ ??? ./configure --prefix=/opt/${ERLANG_VERSION} --without-wx --without-javac --without-odbc --disable-dynamic-ssl-lib && \ ??? make && \ ??? make install && \ ??? rm -rf /tmp/otp_src_${ERLANG_VERSION}* On 06/06/2018 04:29 PM, Vlad Dumitrescu wrote: > Hi all, > > I have a need to rebuild older versions of Erlang on Ubuntu and it > works fine for v20, but for the others I get "no usable openssl > found". I assume that's because a specific version of the libs is > required (what I have is 1.1.0). I need the crypto application > available. How can I make it work??I have little experience in this > area... > > best regards, > Vlad > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- BR, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Jun 6 15:51:19 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 6 Jun 2018 09:51:19 -0400 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <7609E2E2-7FAC-476F-A81C-1C894FA02A60@mac.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> <7609E2E2-7FAC-476F-A81C-1C894FA02A60@mac.com> Message-ID: On Wed, Jun 6, 2018 at 8:56 AM, Sean Hinde wrote: > > "The default Unicode encoding in Erlang is in binaries UTF-8, which is > also the format in which built-in functions and libraries in OTP expect to > find binary Unicode data? > > There is also a strange example in the string.erl document where this > binary <<"abc..????>> is not stored as UTF-8 but instead as latin-1. Having > an unambiguous way to represent a UTF-8 string literal would also clear > this up. > > That seems to point in a clear direction. > > A clarification here. In Erlang, you have to be aware of the following possible encodings: - "abcdef": a string, which is made of straight up unicode codepoints. This means that if you write [16#1f914] you'll quite literally get "??" as a string, with no regards to encoding - <<"abcdef">> as a binary string, which is shorthand for <<$a, $b, $c, $d, $e, $f>>. Which is an old standard list of integers transformed as a binary. By default this format *does not* support unicode encodings, and if you put a value that is too large in there (such as 16#1f914) by declaring a binary like <<"??">>, you will instead find yourself with an overflow, and the final binary <<20>>. - <<"abcdef"/utf8>> as a binary string that is unicode encoded as utf-8. This one would work to support emojis - <<"abcdef"/utf16>> as a binary string that is unicode encoded as utf-16 - <<"abcdef"/utf32>> as a binary string that is unicode encoded as utf-32 - ["abcdef", <<"abcdef"/utf8>>]: iodata type list that can support multiple inputs. As far as I remember, your list can be codepoints as usual, but you'll want all the binaries to be the same encoding (ideally utf-8) to prevent issues where encodings get mixed When the standard library functions say they "expect to find utf-8 by default", it means that when you call functions such as the new ones in the string module, or those in the unicode module where parameters can be given (i.e. unicode:characters_to_binary/1-3), if nothing is specified, then utf-8 is assumed for binaries. But it does not mean that the literal binary strings you write in code are assumed to be utf-8 by default. That's confusing, but yeah. Aside from that, I would say that the choices Elixir made have one risky side to them (the same is possible in Erlang but I'm calling it out because I've seen it a few times in Elixir samples and Erlang has not historically had as many examples of string handling). Because strings are utf8 binaries by default in Elixir, whenever you feel like pattern matching iteratively, you may do something like: <> which in Erlang would be <>. The risk of doing this is that this fetches text by codepoint, whereas when doing text processing, it is often better to do it by grapheme. The best example for that is the family emoji. By default, it could be just a single codepoint, encoded on many bytes, giving: ?? That's fine and good, but the problem comes from the fact that graphical (and logical) representation is not equal to the underlying codes creating the final character. Those exist for all kinds of possible ligatures and assemblies of "character parts" in various languages, but for Emojis, you can also make a family by combining individual people: ??????????? is a family composed of 4 components with combining marks: ?? + ?? + ?? + ??, where + is a special combining mark (a *zero width joiner*) between two women and two boys. If you go ahead and consume that emoji using the /utf8 modifier, you'll break the family apart and change the semantic meaning of the text. If you edit the text in a text editor that traditionally has good support for locales and all kinds of per-language rules, such as Microsoft Word (the only one I know to do a great job of automatically handling half-width spaces and non-breakable spaces when language asks for it), pressing backspace on ??????????? will remove the whole family as one unit. If you do it in FireFox or Chrome, deleting that one 'character' will take you 7 backstrokes: one for each 'person' and one for each zero-width joining character. Slack will consider them to be a single character and visual studio code behaves like the browsers (even if both are electron apps), and notepad.exe or many terminal emulators will instead expand them as 4 people and implicitly drop the zero-width joining marks. If you want to deal with unicode strings, you really should use the string functions from the string module (String in Elixir), and work on graphemes or codepoints depending on the context. One interesting thing there is that you can use these to return you strings re-built up as graphemes using the to_graphemes functions in either language: 1> string:to_graphemes("??e?"). [223,8593,[101,778]]2> string:to_graphemes(<<"??e?"/utf8>>). [223,8593,[101,778]] This lets you take any unicode string, and turn it into a list that is safe to iterate using calls such as lists:map/2 or lists comprehensions. This can only be done through iodata(), and this might even be a better format than what you'd get with just default UTF-8 binary strings. Pattern matching is still risky there. Ideally you'd possibly want to do a round of normalization first, so that characters that can be encoded in more than one way (say ? which can be a single codepoint or a+^ as two points) are forced into a uniform representation. The thing that I'm worried about is how could we make the richness (and pitfalls!) of Unicode handling easier to deal with. So far I've been pleasantly surprised that having no native string type and using codepoints by default did force me to learn a bunch about Unicode and how to do it right, but it's difficult to think that this is the optimal path for everyone. If I had to argue for something, it would be that a good "beginner" string type would be an opaque one that inherently carries its own encoding, and cannot be pattern-matched on unless you use a 'graphemed' + normalized iodata structure. If you wanted to switch to codepoints for handling, then you could convert it to a binary or to another type. But even then this would have a weakness because you would necessarily be forced to convert from say, a utf-8 byte stream coming from a socket, onto a different format: this is exactly what is annoying people today when they just want the damn strings to use "abc" because it's shorter to write. I personally think that this is a clash between correctness and convenience. Currently Erlang is not necessarily 'correct', but it at least does not steer you entirely wrong through convenience since using utf8 (the default many people want) is cumbersome. I'd personally go for a 'correct' option (strongly typed strings that require conversions between formats based on context and usage), but I fear that this thread and most complaints about syntax worry first about convenience, and I don't know that they're easy to reconcile. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Jun 6 15:53:14 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 6 Jun 2018 09:53:14 -0400 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> <7609E2E2-7FAC-476F-A81C-1C894FA02A60@mac.com> Message-ID: On Wed, Jun 6, 2018 at 9:51 AM, Fred Hebert wrote: That's fine and good, but the problem comes from the fact that graphical > (and logical) representation is not equal to the underlying codes creating > the final character. Those exist for all kinds of possible ligatures and > assemblies of "character parts" in various languages, but for Emojis, you > can also make a family by combining individual people: ??????????? is a > family composed of 4 components with combining marks: ?? + ?? + ?? + ??, > where + is a special combining mark (a *zero width joiner*) between two > women and two boys. If you go ahead and consume that emoji using the /utf8 > modifier, you'll break the family apart and change the semantic meaning of > the text. > > If you edit the text in a text editor that traditionally has good support > for locales and all kinds of per-language rules, such as Microsoft Word > (the only one I know to do a great job of automatically handling half-width > spaces and non-breakable spaces when language asks for it), pressing > backspace on ??????????? will remove the whole family as one unit. If you > do it in FireFox or Chrome, deleting that one 'character' will take you 7 > backstrokes: one for each 'person' and one for each zero-width joining > character. Slack will consider them to be a single character and visual > studio code behaves like the browsers (even if both are electron apps), and > notepad.exe or many terminal emulators will instead expand them as 4 people > and implicitly drop the zero-width joining marks. > Well this kind of illustrates my point. Instead of seeing the unicode family as 4 joined people in one unit (see https://emojipedia.org/family-woman-woman-boy-boy/), it appears gmail has expanded the family into 4 distinct people. So please use the emojipedia reference when reading my previous e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Wed Jun 6 15:53:40 2018 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 6 Jun 2018 15:53:40 +0200 Subject: [erlang-questions] Build older versions with right openssl In-Reply-To: References: Message-ID: Great Dmitry, thanks! Now only to check wich libssl version each OTP version requires, and where to get it from :-) best regards, Vlad On Wed, Jun 6, 2018 at 3:40 PM Dmitry Klionsky wrote: > Hi, > > This is how I build R12B5 on CentOS7 in Docker. > > ENV ERLANG_VERSION R12B-5 > > ENV openssl openssl-1.0.2o > > COPY src/${openssl}.tar.gz /tmp/ > RUN cat /tmp/${openssl}.tar.gz | tar xfz - && cd ${openssl} && \ > ./config --prefix=/usr -fpic shared && \ > make && \ > make install && \ > ldconfig && \ > cd .. && rm -rf ${openssl} /tmp/${openssl}.tar.gz > > RUN curl -S -L http://www.erlang.org/download/otp_src_${ERLANG_VERSION}.tar.gz > | tar xz && \ > cd otp_src_${ERLANG_VERSION}/ && \ > ./configure --prefix=/opt/${ERLANG_VERSION} --without-wx > --without-javac --without-odbc --disable-dynamic-ssl-lib && \ > make && \ > make install && \ > rm -rf /tmp/otp_src_${ERLANG_VERSION}* > > > > > > On 06/06/2018 04:29 PM, Vlad Dumitrescu wrote: > > Hi all, > > I have a need to rebuild older versions of Erlang on Ubuntu and it works > fine for v20, but for the others I get "no usable openssl found". I assume > that's because a specific version of the libs is required (what I have is > 1.1.0). I need the crypto application available. How can I make it work? I > have little experience in this area... > > best regards, > Vlad > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -- > BR, > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Wed Jun 6 15:56:18 2018 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 6 Jun 2018 16:56:18 +0300 Subject: [erlang-questions] Build older versions with right openssl In-Reply-To: References: Message-ID: >> Now only to check wich libssl version each OTP version requires, It's try & guess :) >> and where to get it from :-) https://www.openssl.org/source/ On 06/06/2018 04:53 PM, Vlad Dumitrescu wrote: > Great Dmitry, thanks! > > Now only to check wich libssl version each OTP version requires, and > where to get it from :-) > > best regards, > Vlad > > > On Wed, Jun 6, 2018 at 3:40 PM Dmitry Klionsky > wrote: > > Hi, > > This is how I build R12B5 on CentOS7 in Docker. > > ENV ERLANG_VERSION R12B-5 > > ENV openssl openssl-1.0.2o > > COPY src/${openssl}.tar.gz /tmp/ > RUN cat /tmp/${openssl}.tar.gz | tar xfz - && cd ${openssl} && \ > ??? ./config --prefix=/usr -fpic shared && \ > ??? make && \ > ??? make install && \ > ??? ldconfig && \ > ??? cd .. && rm -rf ${openssl} /tmp/${openssl}.tar.gz > > RUN curl -S -L > http://www.erlang.org/download/otp_src_${ERLANG_VERSION}.tar.gz | > tar xz && \ > ??? cd otp_src_${ERLANG_VERSION}/ && \ > ??? ./configure --prefix=/opt/${ERLANG_VERSION} --without-wx > --without-javac --without-odbc --disable-dynamic-ssl-lib && \ > ??? make && \ > ??? make install && \ > ??? rm -rf /tmp/otp_src_${ERLANG_VERSION}* > > > > > > On 06/06/2018 04:29 PM, Vlad Dumitrescu wrote: >> Hi all, >> >> I have a need to rebuild older versions of Erlang on Ubuntu and >> it works fine for v20, but for the others I get "no usable >> openssl found". I assume that's because a specific version of the >> libs is required (what I have is 1.1.0). I need the crypto >> application available. How can I make it work??I have little >> experience in this area... >> >> best regards, >> Vlad >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- > BR, > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- BR, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Jun 6 15:57:53 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 6 Jun 2018 15:57:53 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> Message-ID: <15415c43-3cdc-b82c-0641-51b7a1358170@ninenines.eu> Alternatively, allow making use of Unicode and support the characters ? and ? alongside << and >>. This is already what I use when writing code (using vim's conceal feature) and it makes the source a lot prettier to read. Sames goes for the various arrows. I think another language already does this, was it Ocaml? On 06/05/2018 07:51 PM, Sean Hinde wrote: > I?ve been writing a lot of Elixir over the last few months (plus Swift, Java, C) and just came back to Erlang. There are a few things I?ve come to very much like about Elixir I think might be quite useful to bring to Erlang. > > The first, and topic of this email, is arguably trivial, but having to surround <> string literals with <<>> is irritating and not getting any less so. > > It occurred to me we might usefully steal the sigil notation from Elixir to solve this and add other string literal related syntactic niceties. > > My proposal would be to add an alternative notation for binary string literals in Erlang along the lines of: > > ~s?Some binary string? mapping to <<"Some binary string?>> > > This would open the door to other alternative notations. e.g. Elixir has one for regex friendly strings: > > ~r/foo|bar/ > > String literals containing quotes could also benefit from different delimiter characters e.g. > > ~s'{"Name":"Tom","Age":10}? mapping to <<"{\"Name\":\"Tom\",\"Age\":10}?>> > > The Elixir docs cover more options: https://elixir-lang.org/getting-started/sigils.html. There are 8 alternative delimiters, and even user defined sigils and interpolation with #{} > > If we didn?t want to go as far as Elixir style sigils we could just use a single prefix char. > > LFE uses: > > #?Some binary string? > > or maybe > > ~?Some binary string? > > If people here on the list think there is anything useful in this, and we can reach some kind of rough consensus on how far it makes sense to go I will put some effort into an EEP. > > Sean > > - first post since July 2008. It?s good to be back > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From essen@REDACTED Wed Jun 6 15:58:50 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 6 Jun 2018 15:58:50 +0200 Subject: [erlang-questions] Build older versions with right openssl In-Reply-To: References: Message-ID: <3a3c35b3-172f-878b-f1a4-e559f029137b@ninenines.eu> Most recent 1.0 has been working for me from R16B03 onward. On 06/06/2018 03:53 PM, Vlad Dumitrescu wrote: > Great Dmitry, thanks! > > Now only to check wich libssl version each OTP version requires, and > where to get it from :-) > > best regards, > Vlad > > > On Wed, Jun 6, 2018 at 3:40 PM Dmitry Klionsky > wrote: > > Hi, > > This is how I build R12B5 on CentOS7 in Docker. > > ENV ERLANG_VERSION R12B-5 > > ENV openssl openssl-1.0.2o > > COPY src/${openssl}.tar.gz /tmp/ > RUN cat /tmp/${openssl}.tar.gz | tar xfz - && cd ${openssl} && \ > ??? ./config --prefix=/usr -fpic shared && \ > ??? make && \ > ??? make install && \ > ??? ldconfig && \ > ??? cd .. && rm -rf ${openssl} /tmp/${openssl}.tar.gz > > RUN curl -S -L > http://www.erlang.org/download/otp_src_${ERLANG_VERSION}.tar.gz | > tar xz && \ > ??? cd otp_src_${ERLANG_VERSION}/ && \ > ??? ./configure --prefix=/opt/${ERLANG_VERSION} --without-wx > --without-javac --without-odbc --disable-dynamic-ssl-lib && \ > ??? make && \ > ??? make install && \ > ??? rm -rf /tmp/otp_src_${ERLANG_VERSION}* > > > > > > On 06/06/2018 04:29 PM, Vlad Dumitrescu wrote: >> Hi all, >> >> I have a need to rebuild older versions of Erlang on Ubuntu and it >> works fine for v20, but for the others I get "no usable openssl >> found". I assume that's because a specific version of the libs is >> required (what I have is 1.1.0). I need the crypto application >> available. How can I make it work??I have little experience in >> this area... >> >> best regards, >> Vlad >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- > BR, > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From bryan.hunt@REDACTED Wed Jun 6 16:30:29 2018 From: bryan.hunt@REDACTED (Bryan Hunt) Date: Wed, 6 Jun 2018 15:30:29 +0100 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code Message-ID: Jose, your pragmatic approach is *very* much appreciated. In a top level project, having explicitly chosen to specify the `warning_as_errors` option, a failed build as a result of warnings is sensible. In a dependency, enforcing a top level build failure due to it?s own deprecated code, is at best, unhelpful. While upgrading a project to compile/run on a modern Erlang - it is nearly impossible to avoid the case where one achieves a successful compilation, yet experience a slew of deprecation warnings. I fork all my dependencies in a non-backward-compatibile way to work around this limitation, but this approach is less than desirable from the perspective of cohesion/fragmentation/anything really. Bryan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hinde@REDACTED Wed Jun 6 16:30:46 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Wed, 06 Jun 2018 16:30:46 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> <7609E2E2-7FAC-476F-A81C-1C894FA02A60@mac.com> Message-ID: <81D77CE4-937C-4C99-BBD4-D8D5CBE6D288@mac.com> --- snip - many good and interesting reasons angels fear to tread here --- > > If I had to argue for something, it would be that a good "beginner" string type would be an opaque one that inherently carries its own encoding, and cannot be pattern-matched on unless you use a 'graphemed' + normalized iodata structure. If you wanted to switch to codepoints for handling, then you could convert it to a binary or to another type. But even then this would have a weakness because you would necessarily be forced to convert from say, a utf-8 byte stream coming from a socket, onto a different format: this is exactly what is annoying people today when they just want the damn strings to use "abc" because it's shorter to write. I would fear an even louder chorus if we created a ?beginner? string type that was not useable in most contexts a beginner might want to use a string! Apple have always gone deep on this topic and the solution in Swift is quite ok, at the cost of having to explicitly export to utf-8 to send over the network / store in a file. > > I personally think that this is a clash between correctness and convenience. Currently Erlang is not necessarily 'correct', but it at least does not steer you entirely wrong through convenience since using utf8 (the default many people want) is cumbersome. I'd personally go for a 'correct' option (strongly typed strings that require conversions between formats based on context and usage), but I fear that this thread and most complaints about syntax worry first about convenience, and I don't know that they're easy to reconcile. As an engineering tool for dealing with protocols I would describe Erlang has having tended towards the pragmatic, which could be described as a fine line between correct and convenient. A new notation as a shorthand for utf8 string literals combined with the power of full binary string encoding and lists as code points doesn?t seem like it would be too misleading. Of course in this imaginary language extension we can write any kind of program we like: ~u8?utf-8 string? ~u16?utf-16 string? ~u?unicode string? Though I am with zxq9 that any changes really ought not to make the language worse or less understandable. Rust seems to have got in a mess here: https://github.com/rust-lang/rust/blob/master/src/grammar/raw-string-literal-ambiguity.md Go picked utf8 for literal strings without too much complaint, The slightly wicked side of me would find great enjoyment in a Hacker News post proclaiming Erlang as the one true language for string processing :) Sean From mononcqc@REDACTED Wed Jun 6 17:06:22 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 6 Jun 2018 11:06:22 -0400 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: References: Message-ID: I'm one of the few people who actually at times enforces warnings as errors to dependencies, so that I catch prospective bugs early and can go open pull requests and patch things to make sure I never get surprised with a thing that doesn't build through deprecation. I'd be open to having a mechanism to force-remove it, but I'd be annoyed to lose the ability to go add it back though. On Wed, Jun 6, 2018 at 10:30 AM, Bryan Hunt wrote: > > Jose, your pragmatic approach is *very* much appreciated. > > *In a top level project*, having explicitly *chosen* to specify the > `warning_as_errors` option, > a failed build as a result of warnings is *sensible*. > > *In a dependency*, enforcing a top level build failure due to it?s own *deprecated > code,* > is at best, *unhelpful*. > > While upgrading a project to compile/run on a modern Erlang - it is nearly > impossible to avoid > the case where one achieves a successful compilation, yet experience a > slew of deprecation warnings. > > I fork all my dependencies in a non-backward-compatibile way to work > around this limitation, > but this approach is less than desirable from the perspective of > cohesion/fragmentation/anything really. > > Bryan > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Wed Jun 6 17:14:22 2018 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Wed, 6 Jun 2018 17:14:22 +0200 Subject: [erlang-questions] Deprecation warnings: OTP20 and OTP21 compatible code In-Reply-To: References: Message-ID: > I'd be open to having a mechanism to force-remove it, but I'd be annoyed > to lose the ability to go add it back though. > Maybe have a mechanism that explicitly opts-in to keeping the warnings as errors from dependencies? Something like {warnings_as_errors_from_deps, keep | force | ignore}, but defaults to ignore? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Jun 7 00:21:14 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 07 Jun 2018 07:21:14 +0900 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> Message-ID: <8406409.2YHxOjkTZJ@takoyaki> On 2018?6?6???? 11?41?01? JST Sean Hinde wrote: > As a protocol wrangling language I would argue Erlang has no peers, but many more protocols are string based now than when the bit syntax was invented. By count this is patently false. Most protocols are binary based, as the number of ad hoc binary protocols created for IoT vasty outnumber the handful of prolific string-based ones. Can you think of a better language for IoT protocol wrangling than Erlang? Sure, most people have no clue how to program sockets these days so they use HTTP for everything -- but that isn't *most* protocols, that's a relatively small set of overwhelmingly *prolific* protocols. My prediction is that binary protocols will become more prolific as the extremely limited shared resource of wireless bandwidth becomes more and more saturated (and I don't think compression is a fix-all here, though it certainly helps). It is very hard to tell whether Erlangers as a whole encounter binary or string-based protocols more *often* because a survey on that sort of thing is very hard to pull off -- but it is pretty clear that the sexy, high-profile cases tend to involve the web and that is indeed mostly string-based stuff. There is also the issue with file-format deconstruction (which I do a fair amount of) and a LOT of it is binary. Better handling of UTF-8 (or unicode, more generally, as remember Windows is natively UTF-16...) would be nice as a single case to latch on to and really focus on supporting from every angle -- but it is VERY FAR from being The One Grand Unifying Case. I've commented on quite a few threads about encodings, strings, why graphemes and lexemes matter, and the myopia that comes with dealing with mostly European originated languages. I live in Japan and deal with Shift-JIS, JIS, JIS7, ISO-2022, EUC, etc. variants all the time. Is that a common case? Not in the West, but it is totally normal here -- especially dealing with web data. * Binary protocols are alive and well * The old encodings are far from dead. * You have a good point about improvements being possible and desirable. * The best way to proceed is not clear. * The unicode-correctish improvements to the string and unicode modules are very encouraging. -Craig From zxq9@REDACTED Thu Jun 7 00:48:14 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 07 Jun 2018 07:48:14 +0900 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> Message-ID: <5428620.Bn3D6aMLPc@takoyaki> On 2018?6?6???? 14?00?20? JST Vlad Dumitrescu wrote: > - The new string functions work with strings as sequences of lexemes. The > "list strings" are lists of characters, so for example calling length() on > the two representations of the same text may not return the same value. > Most notably, CRLF is a lexeme, but two characters. To expand on this point (as I've done before) many lexemes used in CJK have multiple constructions that are considered equivalent. Korean hangul is almost a pure example of this, as input is typically done over aggregate lexemes and often re-masked as a single codepoint once the input phase is complete, but not always. Pinyin input works in a similar way but has way more complex aggregate lexemes, though the principle is similar. Even Japanese has a few examples of this (is ? a single character or is it [[?,?]] this particular time we encounter it?). etc. Saying "unicode is the standard now, and UTF-8 is The One True Way" is also saying "a fantastically complex world of codepoints and construction indicators that allow for multiple representations as equivalent is now the standard". That doesn't do anything to solve the question of whether there should be a separate string type. It also ignores that Windows is natively UTF-16, not UTF-8 (though it works a lot better with UTF-8 these days). Go read the unicode standard. It's... well, just have fun reading it. I don't anyone who understands *all* of it 100% -- because for most people the first 10% or so "works for me" (for mainstream CJK use probably the first 30% or so). I suppose all of this is to say that when you compare the enormous number of corner cases in string handling against MERE SYNTAX the syntax is such a trivial issue that it isn't even worth time thinking about. The reason this suject comes up once a year is because we all wish there were a magical set of characters we would write into source files that mean "just make this string work, regardless what it is, because this is hard". We want a syntax that represents the system abstracting away from the underlying data and instead forcing it to mean what *we* mean -- which is insufficiently low-level for much of the work we do as programmers (so we're leaving a lot undefined there, which is bad). At what level do you need to interpret a string? The answer is different for different programs (someone writing a Korean input interpreter is in a very different case than someone writing a chat server). This desire is a kissing cousin of the grand desire for a unified method of l10n and i18n -- which turns out to be just as hard to get right as string handling because every case is different, often in a way that conflicts with another case you have to handle. Blah blah. Strings are hard, and about half of what we think of as "strings" are really serializations of non-string data anyway. -Craig From sean.hinde@REDACTED Thu Jun 7 00:56:29 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Thu, 07 Jun 2018 00:56:29 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <8406409.2YHxOjkTZJ@takoyaki> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> <8406409.2YHxOjkTZJ@takoyaki> Message-ID: > On 7 Jun 2018, at 00:21, zxq9@REDACTED wrote: > > On 2018?6?6???? 11?41?01? JST Sean Hinde wrote: > >> As a protocol wrangling language I would argue Erlang has no peers, but many more protocols are string based now than when the bit syntax was invented. > > By count this is patently false. Most protocols are binary based, as the number of ad hoc binary protocols created for IoT vasty outnumber the handful of prolific string-based ones. Can you think of a better language for IoT protocol wrangling than Erlang? No arguments from me on the suitability of Erlang for protocol wrangling. And these string based ones are definitely prolific. I spent today dealing with json in Erlang for some banking protocol > > Sure, most people have no clue how to program sockets these days so they use HTTP for everything -- but that isn't *most* protocols, that's a relatively small set of overwhelmingly *prolific* protocols. My prediction is that binary protocols will become more prolific as the extremely limited shared resource of wireless bandwidth becomes more and more saturated (and I don't think compression is a fix-all here, though it certainly helps). I don?t think it really matters how we count. Text based protocols are here and Erlang ought to provide a great programming environment for them too. > Better handling of UTF-8 (or unicode, more generally, as remember Windows is natively UTF-16...) would be nice as a single case to latch on to and really focus on supporting from every angle -- but it is VERY FAR from being The One Grand Unifying Case. > > I've commented on quite a few threads about encodings, strings, why graphemes and lexemes matter, and the myopia that comes with dealing with mostly European originated languages. I live in Japan and deal with Shift-JIS, JIS, JIS7, ISO-2022, EUC, etc. variants all the time. Is that a common case? Not in the West, but it is totally normal here -- especially dealing with web data. > > * Binary protocols are alive and well > * The old encodings are far from dead. > * You have a good point about improvements being possible and desirable. > * The best way to proceed is not clear. > * The unicode-correctish improvements to the string and unicode modules are very encouraging. Nice summary. You have obviously thought about this a lot. Any thoughts on a better solution? What would you do? Maybe a hypothetical new string literal type treated as unicode internally but with transparent conversion to utf-8 by default when sent to io (with the option to override)? I get Japan, but utf8 is a sane default. Or maybe some new slick syntax to create a string literal in any encoding. The bit syntax was designed for picking apart bit twiddling telecom protocols. It was clearly not designed with the primary goal of representing alternative forms of string literals. It?s just not what you would choose for that application. Sean > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Thu Jun 7 04:27:34 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 07 Jun 2018 11:27:34 +0900 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <8406409.2YHxOjkTZJ@takoyaki> Message-ID: <6741795.e5IGaYTdQR@takoyaki> On 2018?6?7???? 0?56?29? JST you wrote: > > > On 7 Jun 2018, at 00:21, zxq9@REDACTED wrote: > > > > On 2018?6?6???? 11?41?01? JST Sean Hinde wrote: > > > >> As a protocol wrangling language I would argue Erlang has no peers, but many more protocols are string based now than when the bit syntax was invented. > > > > By count this is patently false. Most protocols are binary based, as the number of ad hoc binary protocols created for IoT vasty outnumber the handful of prolific string-based ones. Can you think of a better language for IoT protocol wrangling than Erlang? > > No arguments from me on the suitability of Erlang for protocol wrangling. And these string based ones are definitely prolific. I spent today dealing with json in Erlang for some banking protocol ... > > * Binary protocols are alive and well > > * The old encodings are far from dead. > > * You have a good point about improvements being possible and desirable. > > * The best way to proceed is not clear. > > * The unicode-correctish improvements to the string and unicode modules are very encouraging. > > Nice summary. You have obviously thought about this a lot. Any thoughts on a better solution? What would you do? > > Maybe a hypothetical new string literal type treated as unicode internally but with transparent conversion to utf-8 by default when sent to io (with the option to override)? I get Japan, but utf8 is a sane default. > > Or maybe some new slick syntax to create a string literal in any encoding. > > The bit syntax was designed for picking apart bit twiddling telecom protocols. It was clearly not designed with the primary goal of representing alternative forms of string literals. It?s just not what you would choose for that application. The main problem I see with this particular example is that you feel you were dealing with a "string-based protocol" because you were dealing with JSON. You weren't -- JSON is a list of trees. It is serialized as a string, and strings are used to represent things in JSON that JSON itself is *dramatically* unsuited for, so "eveything is a string" seems reasonable to people who don't know anything about type systems or were hustled into pushing a "lipstick on a chicken" prototype into production. That last case is so common that a lot of new coders haven't ever seen anything *but* JSON in practice. That doesn't mean we should optimize for wrongness. The point of exacerbation is that you are using a JSON serializer that outputs lists of trees of pairs that contain binary snippets instead of lists as the string representations (Jiffy, I imagine). That isn't the best way to deal with strings in Erlang, imo. So we have a conflation of issues here: - Strings (or more broadly, io_data()) in Erlang can *actually* represent Unicode types because they can represent things as lexemes not just a flat array of codepoints. That's actually quite advanced. - Binaries are just that: binaries. They were indeed never intended for advanced string processing. - Binaries *can* represent strings, are more compact in memory and are easier to deal with in NIFs, which is why Jiffy uses them. - Jiffy is the most common JSON serializer for Erlang. Not a single of these issues is addressed or made easier to deal with by a new syntax that equates to <<"foo"/utf8>>. In fact, the /utf8 binary identifier has only been brought up a few times in this thread because it isn't the point. What you *really* want, I think, is this: 1. A concrete decision about how Erlang represents UTF-8 in memory. A canonicalization. 2. A single io_data() -> utf8_string() IMPORT function. 3. Access to the canonical representation so that dealing with it in Rust/C NIFs and Erlang is not mind bending. 4. A single utf8_string() -> io_data() EXPORT function that has a default serialization rule. 5. A set of functions that allow me to pick which binary representation is output if the default is unsuitable (like when I really need cast hangul characters to their equivalent broken-down lexemes, for example). 6. A special syntax that abstracts the concept of the underlying representation for utf8 in memory. None of these are trivial issues or should be messed about with lightly. As for syntax, quoting we have so far for the types we have so far is great. The <<"blahblah">> thing for direct access to binaries is great. The "foo" == [$f, $o, $o] sugar is also brilliant. The fact that io_data() is a nested list of stuff can very often make complex, large manipulation of io_data() way faster in Erlang than other languages that have to traverse binary strings to do their work, even if it looks ugly (but again, remembering that the *data* you're dealing with is trees merely represented by strings is key). So I think Erlang has really gotten all of that right. But we still SHOULD eventually have a canonical utf8 type. As for syntax... I HATE prefix-glyph syntax for quotes. Ugh. Better to just give me a single-letter function name and let me do u("blah") or whatever. Then I don't have to learn anything new, at least, and can use it in a list function or whatever. I DOUBLY HATE it when new programmes get confused by prefix-glyph syntax. You don't have to teach anyone what a normal-looking quote mark is or how to use or type them. So if we have to have a special syntax, instead, I would recommend backticks-as-quotes. 'an_atom' "a listy string" <<"a binary string">> `a canonical utf8 string` We have a million other kinds of quotes in Japanese that would ?suit??me??just??fine? but totally screw everyone else over, sort of like german quote angle thingies would were they to be made mandatory -- but I think backticks are universally available without any special input modes (correct me if I'm wrong). The `utf8 string` version would be a strict, canonical equivalent to <<"utf8 string"/utf8>> in memory. I'm actually not sure whether the current binary /utf8 tag forces canonicalization (or if it does, *which* unicode form is canonical in Erlang right now). The canonical representation in memory issue has to be ironed out if you want your JSON situation to improve -- and for you I think this is really the rub (whereas I have very different concerns with unicode strings, and would be a bit annoyed if an optimization in the interest of JSON made dealing with things like client-side input or string forms commonly embedded in binary protocol traffic out on my half of the planet unduly complicated). As far as what is happening in Erlang right now to clear some of these issues up, since R19 a LOT of unicode changes have been happening, and most of them are really headed in happy directions. I would say that we need to keep this in the back of our minds, but that implementing anything like unicode canonicalization (to the point that we are happy with whatever is decided forever and ever come-come-what-may-and-screw-the-corner-cases, amen) and especially implementing any special syntax to abstract it in code is premature. Dan Gudmundsson has done a TON of excellent work in this area and continues to do so. He has gained a huge amount of knowledge and experience about unicode and how it interacts with current representations, and he would really be the one to ask about what "should be done" and where we are in terms of reaching a unicode string type that makes sense to deal with internally, in NIFs, exported data, etc. What am I missing? -Craig From essen@REDACTED Thu Jun 7 07:34:00 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 7 Jun 2018 07:34:00 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> <8406409.2YHxOjkTZJ@takoyaki> Message-ID: <5be15c1c-9d73-449d-bfac-f4a957f997b2@ninenines.eu> On 06/07/2018 12:56 AM, Sean Hinde wrote: > > >> On 7 Jun 2018, at 00:21, zxq9@REDACTED wrote: >> >> On 2018?6?6???? 11?41?01? JST Sean Hinde wrote: >> >>> As a protocol wrangling language I would argue Erlang has no peers, but many more protocols are string based now than when the bit syntax was invented. >> >> By count this is patently false. Most protocols are binary based, as the number of ad hoc binary protocols created for IoT vasty outnumber the handful of prolific string-based ones. Can you think of a better language for IoT protocol wrangling than Erlang? > > No arguments from me on the suitability of Erlang for protocol wrangling. And these string based ones are definitely prolific. I spent today dealing with json in Erlang for some banking protocol > >> >> Sure, most people have no clue how to program sockets these days so they use HTTP for everything -- but that isn't *most* protocols, that's a relatively small set of overwhelmingly *prolific* protocols. My prediction is that binary protocols will become more prolific as the extremely limited shared resource of wireless bandwidth becomes more and more saturated (and I don't think compression is a fix-all here, though it certainly helps). > > I don?t think it really matters how we count. Text based protocols are here and Erlang ought to provide a great programming environment for them too. But they're on the way out. You won't find many new text-based protocols, and for good reasons. Even HTTP/2 went binary (and QUIC/HTTP will do the same). Plain-text is still king for content, but the trend has been toward binaries in recent years. Look at the number of binary serialization formats that popped up. Of course, it will be harder to take over JSON. Cheers, -- Lo?c Hoguin https://ninenines.eu From shino.shun@REDACTED Thu Jun 7 07:39:06 2018 From: shino.shun@REDACTED (Shunichi Shinohara) Date: Thu, 7 Jun 2018 14:39:06 +0900 Subject: [erlang-questions] Maybe potential incompatibility of file_info record Message-ID: Hi all, After using OTP 21-rc2 some days, I encountered a situation where rebar3/relx-generated release did not start. The (maybe-indirect) cause is in erlware_commons and the issue has been filed: ec_file:copy fails for not owned source file with OTP 21-rc2 https://github.com/erlware/erlware_commons/issues/133 In the module, file_info is read and written (copied) after file:copy. If source file is owned by root, non-root user can not write the file_info to target's. It seems that some fields are added to file_info record between 20 and 21-rc2, might it be nice if it is stated in OTP release's README as potential incompatibility? Thanks, Shino From bjorn@REDACTED Thu Jun 7 08:04:44 2018 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Thu, 7 Jun 2018 08:04:44 +0200 Subject: [erlang-questions] Maybe potential incompatibility of file_info record In-Reply-To: References: Message-ID: On Thu, Jun 7, 2018 at 7:39 AM, Shunichi Shinohara wrote: [...] > It seems that some fields are added to file_info record between 20 and 21-rc2, "Seems"? No fields have been added for a very long time. The only modifications done in recent releases are updates to the types of the existing fields. The last such change was made at the end of 2015. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From shino.shun@REDACTED Thu Jun 7 08:07:51 2018 From: shino.shun@REDACTED (Shunichi Shinohara) Date: Thu, 7 Jun 2018 15:07:51 +0900 Subject: [erlang-questions] Maybe potential incompatibility of file_info record In-Reply-To: References: Message-ID: I'm sorry I was wrong. Thanks for correct me, Bjorn. I have to dig the root cause down ;) Shino 2018-06-07 15:04 GMT+09:00 Bj?rn Gustavsson : > On Thu, Jun 7, 2018 at 7:39 AM, Shunichi Shinohara wrote: > [...] >> It seems that some fields are added to file_info record between 20 and 21-rc2, > > "Seems"? > > No fields have been added for a very long time. The only > modifications done in recent releases are updates > to the types of the existing fields. The last such change > was made at the end of 2015. > > /Bjorn > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From shino.shun@REDACTED Thu Jun 7 08:52:02 2018 From: shino.shun@REDACTED (Shunichi Shinohara) Date: Thu, 7 Jun 2018 15:52:02 +0900 Subject: [erlang-questions] Maybe potential incompatibility of file_info record In-Reply-To: References: Message-ID: I looked into the source code and I think the difference between 20 and 21.0-rc2 is identified. # The subject of this thread is not suitable, # but please let me report to not leave just a wrong information. First, the difference between OTP 20.3.4 and 21.0-rc2 is as follows. (on macOS, OTPs are installed from git by kerl) Setup file a and b, a is owned by root user. % rm -f a && rm -f b && sudo touch a && sudo chmod 755 a && touch b Let's start OTP 21.0-rc2 and read file_info of a and write it to b, it fails: % ~/local/otp/OTP-21.0-rc2/bin/erl Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-10.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [sharing-preserving] Eshell V10.0 (abort with ^G) 1> {ok, Fi} = file:read_file_info("a"). {ok,#file_info{size = 0,type = regular,access = read, atime = {{2018,6,7},{15,32,41}}, mtime = {{2018,6,7},{15,32,41}}, ctime = {{2018,6,7},{15,32,41}}, mode = 33261,links = 1,major_device = 16777220, minor_device = 0,inode = 17846256,uid = 0,gid = 20}} 2> file:write_file_info("b", Fi). {error,eperm} Then, start OTP 20.3.4 and replay the same step, it succeeds: % ~/local/otp/OTP-20.3.4/bin/erl Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] Eshell V9.3 (abort with ^G) 1> {ok, F} = file:read_file_info("a"). {ok,#file_info{size = 0,type = regular,access = read, atime = {{2018,6,7},{15,32,41}}, mtime = {{2018,6,7},{15,32,41}}, ctime = {{2018,6,7},{15,32,41}}, mode = 33261,links = 1,major_device = 16777220, minor_device = 0,inode = 17846256,uid = 0,gid = 20}} 2> file:write_file_info("b", F). ok The file_info records are the same (sorry again...), but the results differ. Looking into unix_efile.c on maint-20 branch: https://github.com/erlang/otp/blob/maint-20/erts/emulator/drivers/unix/unix_efile.c#L572 it ignores eperm error and proceeds forward: if (chown(name, pInfo->uid, pInfo->gid) && errno != EPERM) { I'm not sure the difference is judged as incompatibility or not, it's nice if this note helps someone (including future me). Thanks, Shino 2018-06-07 15:07 GMT+09:00 Shunichi Shinohara : > I'm sorry I was wrong. Thanks for correct me, Bjorn. > I have to dig the root cause down ;) > > Shino > > 2018-06-07 15:04 GMT+09:00 Bj?rn Gustavsson : >> On Thu, Jun 7, 2018 at 7:39 AM, Shunichi Shinohara wrote: >> [...] >>> It seems that some fields are added to file_info record between 20 and 21-rc2, >> >> "Seems"? >> >> No fields have been added for a very long time. The only >> modifications done in recent releases are updates >> to the types of the existing fields. The last such change >> was made at the end of 2015. >> >> /Bjorn >> >> -- >> Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From ingela.andin@REDACTED Thu Jun 7 09:31:11 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 7 Jun 2018 09:31:11 +0200 Subject: [erlang-questions] Enabling SSL CRL revocation validation for secure URLS in Erlang In-Reply-To: References: Message-ID: Hi! If you expect CRL validation to work smooth please upgrade your Erlang/OTP version. We will consider extending the user guide. For now you can always look at the test suites. Regards Ingela Erlang/OTP team - Ericsson AB 2018-05-30 11:04 GMT+02:00 Soumya Shankar Sardar < soumya.shankar.sardar@REDACTED>: > Hi , > > Need some help on SSL CRL revocation validation enabled for HTTPS in > Erlang code. > > 1) using httpc.requests to access the secure URL. > 2) In the SSL options, we have made {verify:verify_peer} and > {crl_check:peer}. > 3) Also we have added the CRL file in local cache by > ssl_crl_cache:insert(file). CRL file is downloaded from CDP via http. > > Questions > 1) With above setup the CRL validation not failing for revoked URL. Any > idea if the approach is wrong. we followed the erlang.org docs. > 2) Also how we can extend this when there is a CDP[CRL distribution point] > to get the dynamic CRL file. > 3) And how to do above with CDP URL embedded in Server hello message in > the SSL negotiation. > > It will be great to see a sample code with CRL validation in Erlang for > SSL HTTP access. We are using Erlang/OTP 18.1. > > All comments are welcome :) > > > > Regards > > Soumya > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Thu Jun 7 10:35:24 2018 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Thu, 7 Jun 2018 10:35:24 +0200 Subject: [erlang-questions] Maybe potential incompatibility of file_info record In-Reply-To: References: Message-ID: Thanks for digging deeper. The old efile_driver has been rewritten and replaced with a NIF. I am not sure whether this change is an oversight/bug or a deliberate change. We will look into it before the release of OTP 21 and either fix the bug or add a note about the potential incompatibility. /Bj?rn On Thu, Jun 7, 2018 at 8:52 AM, Shunichi Shinohara wrote: > I looked into the source code and I think the difference between 20 > and 21.0-rc2 is identified. > > # The subject of this thread is not suitable, > # but please let me report to not leave just a wrong information. > > First, the difference between OTP 20.3.4 and 21.0-rc2 is as follows. > (on macOS, OTPs are installed from git by kerl) > > Setup file a and b, a is owned by root user. > % rm -f a && rm -f b && sudo touch a && sudo chmod 755 a && touch b > > Let's start OTP 21.0-rc2 and read file_info of a and write it to b, > it fails: > % ~/local/otp/OTP-21.0-rc2/bin/erl > Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-10.0] [source] [64-bit] > [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [sharing-preserving] > > Eshell V10.0 (abort with ^G) > 1> {ok, Fi} = file:read_file_info("a"). > {ok,#file_info{size = 0,type = regular,access = read, > atime = {{2018,6,7},{15,32,41}}, > mtime = {{2018,6,7},{15,32,41}}, > ctime = {{2018,6,7},{15,32,41}}, > mode = 33261,links = 1,major_device = 16777220, > minor_device = 0,inode = 17846256,uid = 0,gid = 20}} > 2> file:write_file_info("b", Fi). > {error,eperm} > > Then, start OTP 20.3.4 and replay the same step, it succeeds: > % ~/local/otp/OTP-20.3.4/bin/erl > Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] > [async-threads:10] [hipe] [kernel-poll:false] > > Eshell V9.3 (abort with ^G) > 1> {ok, F} = file:read_file_info("a"). > {ok,#file_info{size = 0,type = regular,access = read, > atime = {{2018,6,7},{15,32,41}}, > mtime = {{2018,6,7},{15,32,41}}, > ctime = {{2018,6,7},{15,32,41}}, > mode = 33261,links = 1,major_device = 16777220, > minor_device = 0,inode = 17846256,uid = 0,gid = 20}} > 2> file:write_file_info("b", F). > ok > > The file_info records are the same (sorry again...), but the results differ. > > Looking into unix_efile.c on maint-20 branch: > https://github.com/erlang/otp/blob/maint-20/erts/emulator/drivers/unix/unix_efile.c#L572 > it ignores eperm error and proceeds forward: > if (chown(name, pInfo->uid, pInfo->gid) && errno != EPERM) { > > I'm not sure the difference is judged as incompatibility or not, > it's nice if this note helps someone (including future me). > > Thanks, > Shino > > 2018-06-07 15:07 GMT+09:00 Shunichi Shinohara : >> I'm sorry I was wrong. Thanks for correct me, Bjorn. >> I have to dig the root cause down ;) >> >> Shino >> >> 2018-06-07 15:04 GMT+09:00 Bj?rn Gustavsson : >>> On Thu, Jun 7, 2018 at 7:39 AM, Shunichi Shinohara wrote: >>> [...] >>>> It seems that some fields are added to file_info record between 20 and 21-rc2, >>> >>> "Seems"? >>> >>> No fields have been added for a very long time. The only >>> modifications done in recent releases are updates >>> to the types of the existing fields. The last such change >>> was made at the end of 2015. >>> >>> /Bjorn >>> >>> -- >>> Bj?rn Gustavsson, Erlang/OTP, Ericsson AB -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From shino.shun@REDACTED Thu Jun 7 10:38:36 2018 From: shino.shun@REDACTED (Shunichi Shinohara) Date: Thu, 7 Jun 2018 17:38:36 +0900 Subject: [erlang-questions] Maybe potential incompatibility of file_info record In-Reply-To: References: Message-ID: Thanks a lot!! 2018-06-07 17:35 GMT+09:00 Bj?rn Gustavsson : > Thanks for digging deeper. > > The old efile_driver has been rewritten and > replaced with a NIF. I am not sure whether > this change is an oversight/bug or a deliberate > change. We will look into it before the release > of OTP 21 and either fix the bug or add a > note about the potential incompatibility. > > /Bj?rn > > > > On Thu, Jun 7, 2018 at 8:52 AM, Shunichi Shinohara wrote: >> I looked into the source code and I think the difference between 20 >> and 21.0-rc2 is identified. >> >> # The subject of this thread is not suitable, >> # but please let me report to not leave just a wrong information. >> >> First, the difference between OTP 20.3.4 and 21.0-rc2 is as follows. >> (on macOS, OTPs are installed from git by kerl) >> >> Setup file a and b, a is owned by root user. >> % rm -f a && rm -f b && sudo touch a && sudo chmod 755 a && touch b >> >> Let's start OTP 21.0-rc2 and read file_info of a and write it to b, >> it fails: >> % ~/local/otp/OTP-21.0-rc2/bin/erl >> Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-10.0] [source] [64-bit] >> [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [sharing-preserving] >> >> Eshell V10.0 (abort with ^G) >> 1> {ok, Fi} = file:read_file_info("a"). >> {ok,#file_info{size = 0,type = regular,access = read, >> atime = {{2018,6,7},{15,32,41}}, >> mtime = {{2018,6,7},{15,32,41}}, >> ctime = {{2018,6,7},{15,32,41}}, >> mode = 33261,links = 1,major_device = 16777220, >> minor_device = 0,inode = 17846256,uid = 0,gid = 20}} >> 2> file:write_file_info("b", Fi). >> {error,eperm} >> >> Then, start OTP 20.3.4 and replay the same step, it succeeds: >> % ~/local/otp/OTP-20.3.4/bin/erl >> Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] >> [async-threads:10] [hipe] [kernel-poll:false] >> >> Eshell V9.3 (abort with ^G) >> 1> {ok, F} = file:read_file_info("a"). >> {ok,#file_info{size = 0,type = regular,access = read, >> atime = {{2018,6,7},{15,32,41}}, >> mtime = {{2018,6,7},{15,32,41}}, >> ctime = {{2018,6,7},{15,32,41}}, >> mode = 33261,links = 1,major_device = 16777220, >> minor_device = 0,inode = 17846256,uid = 0,gid = 20}} >> 2> file:write_file_info("b", F). >> ok >> >> The file_info records are the same (sorry again...), but the results differ. >> >> Looking into unix_efile.c on maint-20 branch: >> https://github.com/erlang/otp/blob/maint-20/erts/emulator/drivers/unix/unix_efile.c#L572 >> it ignores eperm error and proceeds forward: >> if (chown(name, pInfo->uid, pInfo->gid) && errno != EPERM) { >> >> I'm not sure the difference is judged as incompatibility or not, >> it's nice if this note helps someone (including future me). >> >> Thanks, >> Shino >> >> 2018-06-07 15:07 GMT+09:00 Shunichi Shinohara : >>> I'm sorry I was wrong. Thanks for correct me, Bjorn. >>> I have to dig the root cause down ;) >>> >>> Shino >>> >>> 2018-06-07 15:04 GMT+09:00 Bj?rn Gustavsson : >>>> On Thu, Jun 7, 2018 at 7:39 AM, Shunichi Shinohara wrote: >>>> [...] >>>>> It seems that some fields are added to file_info record between 20 and 21-rc2, >>>> >>>> "Seems"? >>>> >>>> No fields have been added for a very long time. The only >>>> modifications done in recent releases are updates >>>> to the types of the existing fields. The last such change >>>> was made at the end of 2015. >>>> >>>> /Bjorn >>>> >>>> -- >>>> Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > > > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From sean.hinde@REDACTED Thu Jun 7 11:57:26 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Thu, 07 Jun 2018 11:57:26 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <5be15c1c-9d73-449d-bfac-f4a957f997b2@ninenines.eu> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> <8406409.2YHxOjkTZJ@takoyaki> <5be15c1c-9d73-449d-bfac-f4a957f997b2@ninenines.eu> Message-ID: <4F19BB31-8495-4B67-B0C8-2F6A99FF159C@mac.com> >>> Sure, most people have no clue how to program sockets these days so they use HTTP for everything -- but that isn't *most* protocols, that's a relatively small set of overwhelmingly *prolific* protocols. My prediction is that binary protocols will become more prolific as the extremely limited shared resource of wireless bandwidth becomes more and more saturated (and I don't think compression is a fix-all here, though it certainly helps). >> I don?t think it really matters how we count. Text based protocols are here and Erlang ought to provide a great programming environment for them too. > > But they're on the way out. You won't find many new text-based protocols, and for good reasons. Even HTTP/2 went binary (and QUIC/HTTP will do the same). > > Plain-text is still king for content, but the trend has been toward binaries in recent years. Look at the number of binary serialization formats that popped up. Of course, it will be harder to take over JSON. Even as an old school telecom protocols guy I?m not sure I really like this move for web based protocols. It?s nice to be able to read a protocol as text - however good one gets at reading hex dumps. HTTP Headers are a minuscule percentage of internet traffic which is dominated by video. In any case these things tend to have pendulum like properties :) Sean > > Cheers, > > -- > Lo?c Hoguin > https://ninenines.eu From essen@REDACTED Thu Jun 7 12:15:52 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 7 Jun 2018 12:15:52 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <4F19BB31-8495-4B67-B0C8-2F6A99FF159C@mac.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> <8406409.2YHxOjkTZJ@takoyaki> <5be15c1c-9d73-449d-bfac-f4a957f997b2@ninenines.eu> <4F19BB31-8495-4B67-B0C8-2F6A99FF159C@mac.com> Message-ID: <86f6e95a-b62f-b5d3-ea68-296e9044b8b4@ninenines.eu> On 06/07/2018 11:57 AM, Sean Hinde wrote: >>>> Sure, most people have no clue how to program sockets these days so they use HTTP for everything -- but that isn't *most* protocols, that's a relatively small set of overwhelmingly *prolific* protocols. My prediction is that binary protocols will become more prolific as the extremely limited shared resource of wireless bandwidth becomes more and more saturated (and I don't think compression is a fix-all here, though it certainly helps). >>> I don?t think it really matters how we count. Text based protocols are here and Erlang ought to provide a great programming environment for them too. >> >> But they're on the way out. You won't find many new text-based protocols, and for good reasons. Even HTTP/2 went binary (and QUIC/HTTP will do the same). >> >> Plain-text is still king for content, but the trend has been toward binaries in recent years. Look at the number of binary serialization formats that popped up. Of course, it will be harder to take over JSON. > > Even as an old school telecom protocols guy I?m not sure I really like this move for web based protocols. It?s nice to be able to read a protocol as text - however good one gets at reading hex dumps. HTTP Headers are a minuscule percentage of internet traffic which is dominated by video. > > In any case these things tend to have pendulum like properties :) Perhaps. I don't think the "I can read it" argument is a good reason to go one way or the other though. You can read debug, trace or wireshark output just fine. And the protocol being text is not necessarily helpful because of two reasons. One is invisible characters, which requires you to make sure you can see them when debugging anyway (using \n instead of \r\n for example). The other is that on content like JSON, unless your payload is small you're not going to be able to parse it in your head anyway, so you'll need tools to make sense of it. Same reason why developer tools are so useful when writing HTML/CSS. Ultimately I think a language should be good at both, and I think Erlang is doing a fine job at parsing both text and binary. The <<"syntax">> is certainly unfortunate for binary strings, even more so when Unicode is required. I'm not sure this is something that should be fixed at the language level though. Some editors put a ) when you write (, perhaps they should also put >> when you write <<, with or without double quotes. Perhaps they already do? It's a strategy that's worked well for more verbose languages like C++ and Java I think. -- Lo?c Hoguin https://ninenines.eu From sean.hinde@REDACTED Thu Jun 7 12:50:48 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Thu, 07 Jun 2018 12:50:48 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <86f6e95a-b62f-b5d3-ea68-296e9044b8b4@ninenines.eu> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <640C5019-25B9-4057-A4D6-2BE06579A7E9@waisbrot.net> <4BE478B9-924D-45E0-807B-7563F1D5FC7B@mac.com> <8406409.2YHxOjkTZJ@takoyaki> <5be15c1c-9d73-449d-bfac-f4a957f997b2@ninenines.eu> <4F19BB31-8495-4B67-B0C8-2F6A99FF159C@mac.com> <86f6e95a-b62f-b5d3-ea68-296e9044b8b4@ninenines.eu> Message-ID: <0B61B3DE-5D66-477B-BABF-6290C01336D5@mac.com> >> Even as an old school telecom protocols guy I?m not sure I really like this move for web based protocols. It?s nice to be able to read a protocol as text - however good one gets at reading hex dumps. HTTP Headers are a minuscule percentage of internet traffic which is dominated by video. >> In any case these things tend to have pendulum like properties :) > > Perhaps. > > I don't think the "I can read it" argument is a good reason to go one way or the other though. You can read debug, trace or wireshark output just fine. And the protocol being text is not necessarily helpful because of two reasons. > > One is invisible characters, which requires you to make sure you can see them when debugging anyway (using \n instead of \r\n for example). > > The other is that on content like JSON, unless your payload is small you're not going to be able to parse it in your head anyway, so you'll need tools to make sense of it. Same reason why developer tools are so useful when writing HTML/CSS. I recall the ?I can read it? argument was made at the time in part as a rebellion against the "save every bit" mentality in the communications industry standardisation. Encryption also pretty much kills off wireshark unless you have the keys, so we are only talking about end point debugging. But sure, it?s not a big reason. > > Ultimately I think a language should be good at both, and I think Erlang is doing a fine job at parsing both text and binary. The <<"syntax">> is certainly unfortunate for binary strings, even more so when Unicode is required. I'm not sure this is something that should be fixed at the language level though. Some editors put a ) when you write (, perhaps they should also put >> when you write <<, with or without double quotes. Perhaps they already do? It's a strategy that's worked well for more verbose languages like C++ and Java I think. Without reader macros syntax can?t be fixed except at the language level (and no, I?m not arguing we should make another lisp!) But it does feel that when the time is right we ought to be able have the nice things with elegant syntax. BTW The Erlang mode I?m using for VSCode auto adds the closing >> (the one from Pierrick Gourlain, which is really very nice) Also BTW I?m using REST mode of cowboy for this json based system and it?s really very nice. Thank you! Sean > > -- > Lo?c Hoguin > https://ninenines.eu From sean.hinde@REDACTED Thu Jun 7 13:33:58 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Thu, 07 Jun 2018 13:33:58 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <6741795.e5IGaYTdQR@takoyaki> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <8406409.2YHxOjkTZJ@takoyaki> <6741795.e5IGaYTdQR@takoyaki> Message-ID: <48159918-C05E-431B-BBCB-B2C548AA693C@mac.com> >> >> The bit syntax was designed for picking apart bit twiddling telecom protocols. It was clearly not designed with the primary goal of representing alternative forms of string literals. It?s just not what you would choose for that application. > > The main problem I see with this particular example is that you feel you were dealing with a "string-based protocol" because you were dealing with JSON. > > You weren't -- JSON is a list of trees. It is serialized as a string, and strings are used to represent things in JSON that JSON itself is *dramatically* unsuited for, so "eveything is a string" seems reasonable to people who don't know anything about type systems or were hustled into pushing a "lipstick on a chicken" prototype into production. > > That last case is so common that a lot of new coders haven't ever seen anything *but* JSON in practice. That doesn't mean we should optimize for wrongness. > > The point of exacerbation is that you are using a JSON serializer that outputs lists of trees of pairs that contain binary snippets instead of lists as the string representations (Jiffy, I imagine). That isn't the best way to deal with strings in Erlang, imo. That?s a fair summary of my current state of exacerbation :) And yes, jiffy this week. Binary snippets as keys in json suit the use case well enough so long as the decoded representation can be sanely matched against a string literal. Atoms have their other problems, and strings as lists are just plain annoying for dealing with incoming data that can also take the form of lists of other objects (is_string? - https://stackoverflow.com/questions/2479713/determining-if-an-item-is-a-string-or-a-list-in-erlang?noredirect=1&lq=1) > > So we have a conflation of issues here: > - Strings (or more broadly, io_data()) in Erlang can *actually* represent Unicode types because they can represent things as lexemes not just a flat array of codepoints. That's actually quite advanced. > - Binaries are just that: binaries. They were indeed never intended for advanced string processing. > - Binaries *can* represent strings, are more compact in memory and are easier to deal with in NIFs, which is why Jiffy uses them. > - Jiffy is the most common JSON serializer for Erlang. > > Not a single of these issues is addressed or made easier to deal with by a new syntax that equates to <<"foo"/utf8>>. In fact, the /utf8 binary identifier has only been brought up a few times in this thread because it isn't the point. I tend to believe that syntax is important, not for the ?Wah Wah it looks too weird I can?t use that language? reason, but because it defines the UX of the system. And UX does drive behaviour. As much as we would like to think that everyone will think really hard about exactly which representation they need at each point in their program, the current kinds of strings provide a bit of a hobsons choice. > What you *really* want, I think, is this: > 1. A concrete decision about how Erlang represents UTF-8 in memory. A canonicalization. > 2. A single io_data() -> utf8_string() IMPORT function. > 3. Access to the canonical representation so that dealing with it in Rust/C NIFs and Erlang is not mind bending. > 4. A single utf8_string() -> io_data() EXPORT function that has a default serialization rule. > 5. A set of functions that allow me to pick which binary representation is output if the default is unsuitable (like when I really need cast hangul characters to their equivalent broken-down lexemes, for example). > 6. A special syntax that abstracts the concept of the underlying representation for utf8 in memory. If we can have all that without overhead of having to parse byte by byte all incoming data to be sure it?s valid utf8 (utf8_raw mode?) that looks like an excellent way forward. > None of these are trivial issues or should be messed about with lightly. Agreed. The EEP process and culture of this community is well designed to weed out badly thought through proposals :) > > As for syntax, quoting we have so far for the types we have so far is great. The <<"blahblah">> thing for direct access to binaries is great. The "foo" == [$f, $o, $o] sugar is also brilliant. The fact that io_data() is a nested list of stuff can very often make complex, large manipulation of io_data() way faster in Erlang than other languages that have to traverse binary strings to do their work, even if it looks ugly (but again, remembering that the *data* you're dealing with is trees merely represented by strings is key). > > So I think Erlang has really gotten all of that right. > > But we still SHOULD eventually have a canonical utf8 type. > > As for syntax... > > I HATE prefix-glyph syntax for quotes. Ugh. Better to just give me a single-letter function name and let me do u("blah") or whatever. Then I don't have to learn anything new, at least, and can use it in a list function or whatever. > > I DOUBLY HATE it when new programmes get confused by prefix-glyph syntax. You don't have to teach anyone what a normal-looking quote mark is or how to use or type them. > > So if we have to have a special syntax, instead, I would recommend backticks-as-quotes. > > 'an_atom' > "a listy string" > <<"a binary string">> > `a canonical utf8 string` Go-lang made the backtick choice: https://golang.org/ref/spec#String_literals with some interesting sematics. \r chars are stripped out, and their backtick strings can span multiple lines. I?ll do some digging to see how happy their community is with the choice. My only concern would be how easy it could be to mistake ` for ? when reading code. > We have a million other kinds of quotes in Japanese that would ?suit??me??just??fine? but totally screw everyone else over, sort of like german quote angle thingies would were they to be made mandatory -- but I think backticks are universally available without any special input modes (correct me if I'm wrong). At least for programmers it ought to be available - shells have used it forever. > The `utf8 string` version would be a strict, canonical equivalent to <<"utf8 string"/utf8>> in memory. I'm actually not sure whether the current binary /utf8 tag forces canonicalization (or if it does, *which* unicode form is canonical in Erlang right now). The canonical representation in memory issue has to be ironed out if you want your JSON situation to improve -- and for you I think this is really the rub (whereas I have very different concerns with unicode strings, and would be a bit annoyed if an optimization in the interest of JSON made dealing with things like client-side input or string forms commonly embedded in binary protocol traffic out on my half of the planet unduly complicated). JSON handling ought not of course to be the determinant, it?s just this week?s random thing I happen to be working on. I wasn?t working on it last week when the thought came about whether we could steal some ideas from Elixir for string handling. > As far as what is happening in Erlang right now to clear some of these issues up, since R19 a LOT of unicode changes have been happening, and most of them are really headed in happy directions. I would say that we need to keep this in the back of our minds, but that implementing anything like unicode canonicalization (to the point that we are happy with whatever is decided forever and ever come-come-what-may-and-screw-the-corner-cases, amen) and especially implementing any special syntax to abstract it in code is premature. That?s *just* a matter of release planning :) > Dan Gudmundsson has done a TON of excellent work in this area and continues to do so. He has gained a huge amount of knowledge and experience about unicode and how it interacts with current representations, and he would really be the one to ask about what "should be done" and where we are in terms of reaching a unicode string type that makes sense to deal with internally, in NIFs, exported data, etc. Darn, I should have grabbed Dan on this topic at Code BEAM STO last week! > What am I missing? Good question prompted a few more thoughts It would be nice to be able to use a new string format in more of the places we use strings. So some kind of interpolation for string construction: io_lib:format(`~p`, [atom]). to follow existing conventions, or something more modern: `some \(fun() -> ?interpolated? end) string` - Swift or `some #{?interpolated?} string` - Elixir Though without ?printable? protocols I guess these last two wouldn?t fly Thanks for adding your well thought out ideas and views. Sean From jesper.louis.andersen@REDACTED Thu Jun 7 14:29:22 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 7 Jun 2018 14:29:22 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> Message-ID: On Tue, Jun 5, 2018 at 10:57 PM Sean Hinde wrote: > My proposal would be to add an alternative notation for binary string > literals in Erlang along the lines of: > > ~s?Some binary string? mapping to <<"Some binary string?>> > > The underlying problem is that Erlang is chromodynamic, for a lack of better term[0]. In a chromodynamic language, there is one type, term(), but data of that type has "color" insofar data is used with different intent: * ISO8859-15 strings * UTF-8 strings * Lists of integers, where each integer is a code point * binary() payloads * binary() data which has interpretation * bitstring() * integers used as sets of bits And so on. Data is then mapped onto a given subset of term(), namely string(), [non_neg_integer()], [0..255], binary(), iolist(), iodata() etc. Colors don't mix. We can't have green UTF-8 strings together with blue binary() data. But the onus of keeping the colors apart is on the programmer, not on the system. Typed languages (that is the nontrivially typed ones) keeps data apart by means of a type system. So there, we can't mix a UTF-8 string with a binary() blob unless we explicitly convert between the types. However, in a chromodynamic language, we need another way to identify the colors, and this leads into the need for explicit syntactic notation to tell them apart. Worse, our mapping of colorful data to term() is forgetful (or if I may: the mapping is desaturating). So once we have the underlying term(), we don't know from where it came. History plays an important role of course. binary() was intended for binary() data which are just vectors of bytes. But over time, they've found other uses in Erlang systems: * strings() - mostly due to better packing of data. Especially on 64bit machines where list cons cells have considerable overhead. * utf8 encoded strings * dynamic atoms (because Richard O'Keefe's "Split the Atoms proposal was never implemented). You can run out of atoms, but you cannot run out of binary() if you pay the price of more expensive equality checks. Given their prominence, I think it would be good to open a discussion on a more succinct syntax for binary() data. Perhaps laced with a discussion about what utf8 strings should be in the system. Over the years, the ubiquity of binary() data has just slowly grown. Were I the BDFL, I'd probably go with the following scheme: string() - still a list of code points. The double quote is used: "example" binary() - still written as <> atom() - still there, used when you need fast equality checks. I'd probably try to figure out how to GC them so they don't have the current limitation, which would open up their use for more cases where we currently use a binary() text() - A new type specifically for contiguous pieces of unicode. Always encoded as UTF-8. They will have a new syntax, probably `example`. Or perhaps #"example" or ~"example". The latter two has the advantage that they can generalize: ~r".*" etc, but the former is awfully readable. This introduces a honest-to-god type for textual data where the data is processed as a whole, and it would probably accept a fully backwards compatible representation. We need to discriminate between binary() and textual data at the lowest level anyway. Otherwise you run the risk of mixing color way too often. Conversion routines should verify and crash on conversions which are not allowed. Rationale: I'd never create the string() type were I to create a new language. A string is not defined as a list of codepoints, but rather as a vector of codepoints (which also means they are immutable). They should support O(1) concatenation (by having the internal representation be either iodata() or a finger tree). But since we have so much legacy code, we are stuck with string(), much like Haskell is where String = [Char]. End of BFDL rant :) [0] In keeping with CS tradition, I'll take a term from physics and absolutely butcher it by using it in a different context where it doesn't belong. Bear with me, for I have sinned. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Thu Jun 7 16:11:58 2018 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Thu, 7 Jun 2018 10:11:58 -0400 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <5428620.Bn3D6aMLPc@takoyaki> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <5428620.Bn3D6aMLPc@takoyaki> Message-ID: > Saying "unicode is the standard now, and UTF-8 is The One True Way" is also saying > "a fantastically complex world of codepoints and construction indicators that allow > for multiple representations as equivalent is now the standard". That doesn't do > anything to solve the question of whether there should be a separate string type. It > also ignores that Windows is natively UTF-16, not UTF-8 (though it works a lot better > with UTF-8 these days). Hi Craig, I?d like to say ?right on!,? but I probably shouldn?t participate in this debate. For one, I?m not a professional programmer. I?ve only, painfully, worked hard to learn Erlang to solve a very specific problem. So I bring a pragmatic beginner?s mind to this discussion that all are free to discount. And, as an English speaker, I bring an ashamedly provincial bias. Indeed, after seventy some odd years I still struggle to express myself fluently in English. My pain point is this: I cringe now every time I want to use an Erlang string function. Since my aging memory is not now what it once was, I need to consult the reference manual frequently while I?m programming. And, I must admit, that the new string functions baffle, frustrate and, unreasonably, enrage me. I totally lose flow and concentration when I need to do what was once the simplest string operation, spending many many precious minutes trying to understand the new and improved way of going about it. So, just a few observations for what they?re worth: Seems to me that trying to find the one universal digital standard for representing all the wonderful organic and evolving natural languages in the world is an exercise in hubris. There is no surer road to complexity and bloat than trying to be all things to all people. But, yes, in this global world, we do need to communicate across natural language domains. Esperanto is one not terribly successful attempt to do this in the non-digital world. How many among us speak Esperanto? Yet, we use translators and translation services quite effectively. So, perhaps, the glyphs of each language should have their own most efficient and standardized digital representation. And serious intellectual capital should go into writing language-to-language translation packages. All the best to all, LRP Sent from my iPad > On Jun 6, 2018, at 6:48 PM, zxq9@REDACTED wrote: > >> On 2018?6?6???? 14?00?20? JST Vlad Dumitrescu wrote: >> >> - The new string functions work with strings as sequences of lexemes. The >> "list strings" are lists of characters, so for example calling length() on >> the two representations of the same text may not return the same value. >> Most notably, CRLF is a lexeme, but two characters. > > To expand on this point (as I've done before) many lexemes used in CJK have multiple constructions that are considered equivalent. Korean hangul is almost a pure example of this, as input is typically done over aggregate lexemes and often re-masked as a single codepoint once the input phase is complete, but not always. Pinyin input works in a similar way but has way more complex aggregate lexemes, though the principle is similar. Even Japanese has a few examples of this (is ? a single character or is it [[?,?]] this particular time we encounter it?). > > etc. > > Saying "unicode is the standard now, and UTF-8 is The One True Way" is also saying "a fantastically complex world of codepoints and construction indicators that allow for multiple representations as equivalent is now the standard". That doesn't do anything to solve the question of whether there should be a separate string type. It also ignores that Windows is natively UTF-16, not UTF-8 (though it works a lot better with UTF-8 these days). > > Go read the unicode standard. It's... well, just have fun reading it. I don't anyone who understands *all* of it 100% -- because for most people the first 10% or so "works for me" (for mainstream CJK use probably the first 30% or so). > > I suppose all of this is to say that when you compare the enormous number of corner cases in string handling against MERE SYNTAX the syntax is such a trivial issue that it isn't even worth time thinking about. The reason this suject comes up once a year is because we all wish there were a magical set of characters we would write into source files that mean "just make this string work, regardless what it is, because this is hard". We want a syntax that represents the system abstracting away from the underlying data and instead forcing it to mean what *we* mean -- which is insufficiently low-level for much of the work we do as programmers (so we're leaving a lot undefined there, which is bad). At what level do you need to interpret a string? The answer is different for different programs (someone writing a Korean input interpreter is in a very different case than someone writing a chat server). This desire is a kissing cousin of the grand desire for a unified method of l10n and i18n -- which turns out to be just as hard to get right as string handling because every case is different, often in a way that conflicts with another case you have to handle. > > Blah blah. Strings are hard, and about half of what we think of as "strings" are really serializations of non-string data anyway. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From leonard.boyce@REDACTED Fri Jun 8 15:03:15 2018 From: leonard.boyce@REDACTED (Leonard B) Date: Fri, 8 Jun 2018 09:03:15 -0400 Subject: [erlang-questions] SFTP Server and state Message-ID: Hoping someone has done something similar before and can point me in the right direction. I'm looking to implement a virtual file system behind sftpd using a custom file handler. This is all well and good, as you can easily specify the subsystem using {subsystems, ssh_sftpd:subsystem_spec([{file_handler, my_file_handler}])} Where I'm running into an issue is that there appears to be no way to connect the subsystem to the auth handler If I specify pwdfun I can handle my DB based auth without issue, but the file subsystem has no access to the data. Reading the code for ssh_sftp:init/1 it appears that the {ssh_sftpd_file, _} option can accept a tuple of {module, FileHandlerState}. The issue I have is how to get the `pwdfun` state (optional return) into the file subsystem. Any pointers would be great. Thanks, Leonard From attila.r.nohl@REDACTED Fri Jun 8 15:56:47 2018 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Fri, 8 Jun 2018 14:56:47 +0100 Subject: [erlang-questions] ct:pal crashing Message-ID: Recently our common_test suites started to fail in our (gitlab-runner) CI environment with errors like this: ????????{dir_SUITE,init_per_testcase, ????????????{{noproc, ???????????????? {gen_event,call, ???????????????????? [cth_readable_logger,cth_readable_failonly, ??????????????????????{ct_pal, ??????????????????????????[10,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, ?????????????????????????? 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, ?????????????????????????? 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, ?????????????????????????? 45,45,10,"2018-06-06 19:41:49.002",10,116,101,115, ?????????????????????????? 116,32,105,110,105,116,32,39,"bic",39,58,32,68,101, ?????????????????????????? 108,101,116,101,100,32,107,101,121,115,32,"[]",10, ?????????????????????????? 10]}]}}, ???????????? [{gen_event,call1,3,[{file,"gen_event.erl"},{line,207}]}, ??????????????{cthr,pal,4, ??????????????????[{file, "/usr/src/rebar3-src/_build/default/lib/cth_readable/src/cthr.erl"}, ?????????????????? {line,74}]}, As far as I understand the gitlab CI environment and our setup, the CI job fetches the erlang:19 Docker image: Using docker image sha256:2ea95f0c3147e50b79b4fcd28c955c80c2e57eb20fb8ff61bb8ed1873d0b6957 for erlang:19 and executes our test suite in that image. Our code did not change, but it looks like the image did change, at least at the latest successful test I see this in the logs: Using docker image sha256:e36b5300c4cf2f504f0222ea66bbf81fb2e514ad88cb93f111b5e85e7893b99a for erlang:19 Do you know what could cause this problem? From mononcqc@REDACTED Fri Jun 8 17:57:33 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 8 Jun 2018 11:57:33 -0400 Subject: [erlang-questions] ct:pal crashing In-Reply-To: References: Message-ID: That error is due to https://github.com/ferd/cth_readable/, a CT hook that rebar3 bundles to make CT output look nicer. Usually this is stable stuff that worked well for multiple versions, and the only broken one I'm aware of is OTP-21-rc2 (the fix is in master, but hasn't been put in an official release yet). The error makes it look like the following code is being called: https://github.com/ferd/cth_readable/blob/master/src/cthr.erl#L74-L75 This is dynamic detection of the logger module in OTP-21, which requires output to be redirected in another place than the default error logger. If your own project happens to also include a module named 'logger', then you're getting into a clash with code that attempts to support a newer version of Erlang. If that is what happens, I can only advise to: - possibly disable the pretty output by adding --readable=false to your command, or {ct_readable, false} to your rebar.config file - switch to an older rebar3 version that will support only the older logging interface - explicitly disable ct_readable, manually include an older version of cth_readable and use that one instead by specifying CT hooks in {ct_opts, [{hooks, [...]}]} If you do not have a logger module in your OTP-19 version, then I'm going to need more details to figure that one out. On Fri, Jun 8, 2018 at 9:56 AM, Attila Rajmund Nohl wrote: > Recently our common_test suites started to fail in our (gitlab-runner) > CI environment with errors like this: > > {dir_SUITE,init_per_testcase, > {{noproc, > {gen_event,call, > [cth_readable_logger,cth_readable_failonly, > {ct_pal, > [10,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, > 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, > 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, > 45,45,10,"2018-06-06 19:41:49.002",10,116,101,115, > 116,32,105,110,105,116,32,39,"bic",39,58,32,68,101, > 108,101,116,101,100,32,107,101,121,115,32,"[]",10, > 10]}]}}, > [{gen_event,call1,3,[{file,"gen_event.erl"},{line,207}]}, > {cthr,pal,4, > [{file, > > "/usr/src/rebar3-src/_build/default/lib/cth_readable/src/cthr.erl"}, > {line,74}]}, > > As far as I understand the gitlab CI environment and our setup, the CI > job fetches the erlang:19 Docker image: > > Using docker image > sha256:2ea95f0c3147e50b79b4fcd28c955c80c2e57eb20fb8ff61bb8ed1873d0b6957 > for erlang:19 > > and executes our test suite in that image. Our code did not change, > but it looks like the image did change, at least at the latest > successful test I see this in the logs: > > Using docker image > sha256:e36b5300c4cf2f504f0222ea66bbf81fb2e514ad88cb93f111b5e85e7893b99a > for erlang:19 > > Do you know what could cause this problem? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila.r.nohl@REDACTED Fri Jun 8 18:32:06 2018 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Fri, 8 Jun 2018 17:32:06 +0100 Subject: [erlang-questions] ct:pal crashing In-Reply-To: References: Message-ID: Thanks. We do have a logger module that's in dire need of renaming. Fred Hebert ezt ?rta (id?pont: 2018. j?n. 8., P?n 16:58): > That error is due to https://github.com/ferd/cth_readable/, a CT hook > that rebar3 bundles to make CT output look nicer. > > Usually this is stable stuff that worked well for multiple versions, and > the only broken one I'm aware of is OTP-21-rc2 (the fix is in master, but > hasn't been put in an official release yet). The error makes it look like > the following code is being called: > > https://github.com/ferd/cth_readable/blob/master/src/cthr.erl#L74-L75 > > This is dynamic detection of the logger module in OTP-21, which requires > output to be redirected in another place than the default error logger. If > your own project happens to also include a module named 'logger', then > you're getting into a clash with code that attempts to support a newer > version of Erlang. > > If that is what happens, I can only advise to: > > - possibly disable the pretty output by adding --readable=false to > your command, or {ct_readable, false} to your rebar.config file > - switch to an older rebar3 version that will support only the older > logging interface > - explicitly disable ct_readable, manually include an older version of > cth_readable and use that one instead by specifying CT hooks in {ct_opts, > [{hooks, [...]}]} > > If you do not have a logger module in your OTP-19 version, then I'm going > to need more details to figure that one out. > > > On Fri, Jun 8, 2018 at 9:56 AM, Attila Rajmund Nohl < > attila.r.nohl@REDACTED> wrote: > >> Recently our common_test suites started to fail in our (gitlab-runner) >> CI environment with errors like this: >> >> {dir_SUITE,init_per_testcase, >> {{noproc, >> {gen_event,call, >> [cth_readable_logger,cth_readable_failonly, >> {ct_pal, >> [10,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, >> 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, >> 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, >> 45,45,10,"2018-06-06 19:41:49.002",10,116,101,115, >> 116,32,105,110,105,116,32,39,"bic",39,58,32,68,101, >> 108,101,116,101,100,32,107,101,121,115,32,"[]",10, >> 10]}]}}, >> [{gen_event,call1,3,[{file,"gen_event.erl"},{line,207}]}, >> {cthr,pal,4, >> [{file, >> >> "/usr/src/rebar3-src/_build/default/lib/cth_readable/src/cthr.erl"}, >> {line,74}]}, >> >> As far as I understand the gitlab CI environment and our setup, the CI >> job fetches the erlang:19 Docker image: >> >> Using docker image >> sha256:2ea95f0c3147e50b79b4fcd28c955c80c2e57eb20fb8ff61bb8ed1873d0b6957 >> for erlang:19 >> >> and executes our test suite in that image. Our code did not change, >> but it looks like the image did change, at least at the latest >> successful test I see this in the logs: >> >> Using docker image >> sha256:e36b5300c4cf2f504f0222ea66bbf81fb2e514ad88cb93f111b5e85e7893b99a >> for erlang:19 >> >> Do you know what could cause this problem? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From list1@REDACTED Sun Jun 10 17:07:08 2018 From: list1@REDACTED (Grzegorz Junka) Date: Sun, 10 Jun 2018 15:07:08 +0000 Subject: [erlang-questions] RFC The Computer Nonsense Guide In-Reply-To: References: Message-ID: On 01/06/2018 21:28, Jean Chassoul wrote: > Hope you all have fun on Stockholm this week, > > and now for something completely different: a request for comments and > feedback from the Latin American jungle. > > https://github.com/nonsensews/guide/wiki > > Jean Hi Jean, After spending a few minutes I still don't know what this guide is trying to guide me to. Is this a joke? Some existential narrative? A software? A framework? An idea? Do I need it? Why should I care? I would need to know what you are trying to introduce before I can decide if I would have enough expertise or interest to evaluate the guide and of course if I would want to invest more than those few minutes to read it more fully. That would my feedback so far. Thanks GrzegorzJ From jchassoul@REDACTED Mon Jun 11 00:12:01 2018 From: jchassoul@REDACTED (Jean Chassoul) Date: Sun, 10 Jun 2018 18:12:01 -0400 Subject: [erlang-questions] RFC The Computer Nonsense Guide In-Reply-To: References: Message-ID: Hi Grzegorz, Thanks for stop for a few minutes into the guide, The work will perhaps only be understood by those who have themselves already thought the thoughts which are expressed in it, or similar thoughts. It is therefore not a guidebook maybe a gateway?. Its objective would be attained if it afforded pleasure to one who read it with understanding. Indeed what I have here makes no claim to novelty in points or details; and that is why I give you no sources, if the bits have value it consists in that thoughts are expressed in it, and this value will be the greater the better the thoughts are expressed. Thanks for your email and the feedback. Jean On Sun, Jun 10, 2018 at 11:07 AM, Grzegorz Junka wrote: > > On 01/06/2018 21:28, Jean Chassoul wrote: > >> Hope you all have fun on Stockholm this week, >> >> and now for something completely different: a request for comments and >> feedback from the Latin American jungle. >> >> https://github.com/nonsensews/guide/wiki >> >> Jean >> > > Hi Jean, > After spending a few minutes I still don't know what this guide is trying > to guide me to. Is this a joke? Some existential narrative? A software? A > framework? An idea? Do I need it? Why should I care? > I would need to know what you are trying to introduce before I can decide > if I would have enough expertise or interest to evaluate the guide and of > course if I would want to invest more than those few minutes to read it > more fully. > That would my feedback so far. > Thanks > GrzegorzJ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fchschneider@REDACTED Mon Jun 11 09:43:02 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Mon, 11 Jun 2018 09:43:02 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state Message-ID: Dear list, the state_enter_result(State) has as one its return values {next_state, State, NewData, ...} but explicitly disallows the State to be different from the current State,? First, I find it confusing to allow 'next_state' here since State cannot change but the name makes the suggestion it can. Secondly, I would love to be able to actually make a state change. Quite often i find myself running a few tests on entering a state and making state changes based on these tests. Now I have to move these tests to all states which can have this state as a result and go directly to the end state. Would it be possible to change state from a state enter call? Frans From sean.hinde@REDACTED Mon Jun 11 10:22:43 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Mon, 11 Jun 2018 10:22:43 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> Message-ID: <4C1BF0DD-3AE3-4F75-9A7D-9130F7509D41@mac.com> Seems like a good time for a summary. Yell if I have misunderstood or mis-represented - There is little appetite for a new ?bandaid' syntax mapping to <> - Erlang is already on a path towards much improved string handling (although at the cost of making the simple cases more complex?) - No-one is arguing we have the ideal solution for strings today - There is appetite for a new string type. This should have a distinct dynamic type - text() ? - This type ought to be represented internally in a carefully chosen canonical utf-8 format - It needs clean and efficient mechanisms for input and output to file / network / nif with utf-8 default but option for many other representations. - Backtick quotes have a few votes as new syntax for such a representation. Go-lang has chosen backtick for their raw strings with some interesting semantics. - A new text() type would allow io to print these strings as `utf8 string` rather than falling back to binary representation. Some issues: - Adding a new type raises all the usual questions about equality, ordering, conversion (implicit and via a many to many matrix of conversion functions), guards etc. It?s a much larger change than a simple syntax representation mapping to <<"some string"/utf8>> - How do we concatenate them? `Hello` <> `World`? - How do we construct them? io_lib:format(`~p`, [atom]). `hello #{Name}` ? - how do we incorporate them in other string types? io:format(?~s?, [`text`]). <> - How do we extract them from binary data? <>. What is the meaning of the length parameter? The string module already has a clear definition. - What does matching a literal out of binary data mean? <<`H?ll?`, Rest/text>> == <> - Prefix matching in normal code? `Hello` <> World = `Hello world` - Is there already a suitable internal canonical utf-8 format? OTP team? - Lots of other details in the mails from everyone - Everything else Sigils are orthogonal to this discussion (and one of the secondary benefits is pretty nicely realised by using backtick - the more common double quotes would not need to be escaped - yes, JSON). So,the big question: Did this already reach a level of complexity the language will sink under? Is it worth spending time fleshing any of this out to an EEP level of detail? Do we come back in another year? Would an EEP help the existing work of the OTP team in this area or is there already a clear plan and this would be a distraction? Sean Aside: We don?t have the BDFL question. Instead Erlang/OTP has a process One old boss I respected explained a reason big companies like to buy from big companies even at many times the price - his reason was that small companies rely on people and big companies on process. The process will deliver (eventually) even if the people change three times in the middle! > On 7 Jun 2018, at 14:29, Jesper Louis Andersen wrote: > > On Tue, Jun 5, 2018 at 10:57 PM Sean Hinde > wrote: > My proposal would be to add an alternative notation for binary string literals in Erlang along the lines of: > > ~s?Some binary string? mapping to <<"Some binary string?>> > > > The underlying problem is that Erlang is chromodynamic, for a lack of better term[0]. In a chromodynamic language, there is one type, term(), but data of that type has "color" insofar data is used with different intent: > > * ISO8859-15 strings > * UTF-8 strings > * Lists of integers, where each integer is a code point > * binary() payloads > * binary() data which has interpretation > * bitstring() > * integers used as sets of bits > > And so on. Data is then mapped onto a given subset of term(), namely string(), [non_neg_integer()], [0..255], binary(), iolist(), iodata() etc. > > Colors don't mix. We can't have green UTF-8 strings together with blue binary() data. But the onus of keeping the colors apart is on the programmer, not on the system. > > Typed languages (that is the nontrivially typed ones) keeps data apart by means of a type system. So there, we can't mix a UTF-8 string with a binary() blob unless we explicitly convert between the types. However, in a chromodynamic language, we need another way to identify the colors, and this leads into the need for explicit syntactic notation to tell them apart. > > Worse, our mapping of colorful data to term() is forgetful (or if I may: the mapping is desaturating). So once we have the underlying term(), we don't know from where it came. > > History plays an important role of course. binary() was intended for binary() data which are just vectors of bytes. But over time, they've found other uses in Erlang systems: > > * strings() - mostly due to better packing of data. Especially on 64bit machines where list cons cells have considerable overhead. > * utf8 encoded strings > * dynamic atoms (because Richard O'Keefe's "Split the Atoms proposal was never implemented). You can run out of atoms, but you cannot run out of binary() if you pay the price of more expensive equality checks. > > Given their prominence, I think it would be good to open a discussion on a more succinct syntax for binary() data. Perhaps laced with a discussion about what utf8 strings should be in the system. Over the years, the ubiquity of binary() data has just slowly grown. > > Were I the BDFL, I'd probably go with the following scheme: > > string() - still a list of code points. The double quote is used: "example" > binary() - still written as <> > atom() - still there, used when you need fast equality checks. I'd probably try to figure out how to GC them so they don't have the current limitation, which would open up their use for more cases where we currently use a binary() > text() - A new type specifically for contiguous pieces of unicode. Always encoded as UTF-8. They will have a new syntax, probably `example`. Or perhaps #"example" or ~"example". The latter two has the advantage that they can generalize: ~r".*" etc, but the former is awfully readable. > > This introduces a honest-to-god type for textual data where the data is processed as a whole, and it would probably accept a fully backwards compatible representation. We need to discriminate between binary() and textual data at the lowest level anyway. Otherwise you run the risk of mixing color way too often. Conversion routines should verify and crash on conversions which are not allowed. > > Rationale: I'd never create the string() type were I to create a new language. A string is not defined as a list of codepoints, but rather as a vector of codepoints (which also means they are immutable). They should support O(1) concatenation (by having the internal representation be either iodata() or a finger tree). But since we have so much legacy code, we are stuck with string(), much like Haskell is where String = [Char]. > > End of BFDL rant :) > > > [0] In keeping with CS tradition, I'll take a term from physics and absolutely butcher it by using it in a different context where it doesn't belong. Bear with me, for I have sinned. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Mon Jun 11 10:45:33 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 11 Jun 2018 10:45:33 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: References: Message-ID: <20180611084533.GA61770@erix.ericsson.se> On Mon, Jun 11, 2018 at 09:43:02AM +0200, Frans Schneider wrote: > Dear list, > > the state_enter_result(State) has as one its return values {next_state, > State, NewData, ...} but explicitly disallows the State to be different > from the current State,? First, I find it confusing to allow > 'next_state' here since State cannot change but the name makes the Throughout gen_statem is is by design so that there is no difference between {next_state, SameState, NewData, ...} and {keep_state, NewData, ...}. It is {keep_state, ...} that is shorthand for {next_state, SameState, ...}. So dissalowing {next_state, ...} in a state_enter call would be an exception to the main rule that {next_state, ...} is the generic form. There might be situations where you have a helper function that calculates the next state so it would want to always return {next_state, ...}, and then you want to use it in a state_enter call context knowing it will always return {next_state, SameState, ...}. I think this use should be possible. > suggestion it can. Secondly, I would love to be able to actually make a > state change. Quite often i find myself running a few tests on entering > a state and making state changes based on these tests. Now I have to The state_enter call is intended for code to be executed when entering a state, making this code co-located in the source with the state's event handling code. If the state_enter call should be allowed to change states it would no longer be co-located since it would for example start a timer for the other state it changes to. The alternative to using state_enter calls would be to have a convention where you at exit from the other state go through a helper function that does what you need for state entry, and remember to use this function for every state entry to the state in question. I think your use case is a good example for when having a specialized state change function makes sense. There is no single state this kind of code clearly should be co-located with. > move these tests to all states which can have this state as a result and > go directly to the end state. > > Would it be possible to change state from a state enter call? I do not like that idea, for the reasons above... > > Frans -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From fchschneider@REDACTED Mon Jun 11 13:07:31 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Mon, 11 Jun 2018 13:07:31 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180611084533.GA61770@erix.ericsson.se> References: <20180611084533.GA61770@erix.ericsson.se> Message-ID: Thanks Raimo, See your point, but I am not yet really convinced. Using a helper function would in my case.opinion make the code more obliterative. I think the following code makes it very clear what goes on while some helper function called from whatever other place would make things much more obscure. handle_event(enter, _Old_state, #state{reliability_level = reliable, state1 = announcing} = State, #data{changes = []} = Data) -> {next_state, State#state{state1 = idle}, Data}; handle_event(enter, _Old_state, #state{reliability_level = reliable, state1 = announcing}, #data{push_mode = true} = Data -> {next_state, State#state{state1 = pushing}, Data}; handle_event(enter, _Old_state, #state{reliability_level = reliable, state1 = announcing}, #data{push_mode = false, heartbeat_period = Heartbeat_period} = Data) -> Hb_timer_ref = erlang:start_timer(Heartbeat_period, self(), hb), {keep_state, Data#data{heartbeat_timer_ref = Hb_timer_ref}}; Currently, I resort to testing for a particular condition in the state enter code and insert a next event (changes_not_empty) to trigger the state change I need, which however also introduces extra code and makes things also less obvious. Also, your co-locating argument is arguable since when using complex states and handle_events, I tend to use a different ordering of the functions, mostly focusing on the type of events to handle. (I know, 'enter' isn't an event of course.) Frans On 06/11/2018 10:45 AM, Raimo Niskanen wrote: > On Mon, Jun 11, 2018 at 09:43:02AM +0200, Frans Schneider wrote: >> Dear list, >> >> the state_enter_result(State) has as one its return values {next_state, >> State, NewData, ...} but explicitly disallows the State to be different >> from the current State,? First, I find it confusing to allow >> 'next_state' here since State cannot change but the name makes the > > Throughout gen_statem is is by design so that there is no difference > between {next_state, SameState, NewData, ...} > and {keep_state, NewData, ...}. It is {keep_state, ...} that is shorthand > for {next_state, SameState, ...}. > > So dissalowing {next_state, ...} in a state_enter call would be an > exception to the main rule that {next_state, ...} is the generic form. > > There might be situations where you have a helper function that calculates > the next state so it would want to always return {next_state, ...}, and > then you want to use it in a state_enter call context knowing it will > always return {next_state, SameState, ...}. I think this use should be > possible. > >> suggestion it can. Secondly, I would love to be able to actually make a >> state change. Quite often i find myself running a few tests on entering >> a state and making state changes based on these tests. Now I have to > > The state_enter call is intended for code to be executed when entering a > state, making this code co-located in the source with the state's event > handling code. > > If the state_enter call should be allowed to change states it would no > longer be co-located since it would for example start a timer for the other > state it changes to. > > The alternative to using state_enter calls would be to have a convention > where you at exit from the other state go through a helper function > that does what you need for state entry, and remember to use this > function for every state entry to the state in question. > > I think your use case is a good example for when having a specialized state > change function makes sense. There is no single state this kind of code > clearly should be co-located with. > >> move these tests to all states which can have this state as a result and >> go directly to the end state. >> >> Would it be possible to change state from a state enter call? > > I do not like that idea, for the reasons above... > >> >> Frans > From natalia.chechina@REDACTED Mon Jun 11 16:22:28 2018 From: natalia.chechina@REDACTED (Natalia Chechina) Date: Mon, 11 Jun 2018 15:22:28 +0100 Subject: [erlang-questions] [Last CfP]: Erlang Workshop -- St. Louis, Missouri, US, 29 Sept 2018 -- Deadline: 15 June 2018 Message-ID: Dear all, This is the last call for paper submissions for Erlang Workshop 2018. Deadline is the **15th of June 2018**. Please, consider submitting your work. Erlang Workshop 2018 will take place in St. Louis, Missouri, US on the 29th of Sept 2018. Technical, practice, and application papers related to Erlang, BEAM, Elixir, Scala/Akka, CloudHaskell, Lisp Flavoured Erlang, OCaml, and functional programming in general are welcome and encouraged. Max length for full papers is 12pages, for short ? 6pages. Looking forward to receiving your papers, Erlang Workshop 2018 co-chairs Natalia and Adrian. =================== LAST CALL FOR PAPERS =================== Seventeenth ACM SIGPLAN Erlang Workshop http://erlang.org/workshop/2018/ ----------------------------------------------------------- St. Louis, Missouri, United States, 29 September 2018 Satellite event of the 23rd ACM SIGPLAN International Conference on Functional Programming (ICFP 2018) 23 - 29 September, 2018 The Erlang Workshop aims to bring together the open source, academic, and industrial communities of Erlang, to discuss technologies and languages related to Erlang. The Erlang model of concurrent programming has been widely emulated, for example by Akka in Scala, and even new programming languages were designed atop of the Erlang VM, such as Elixir. Therefore we would like to broaden the scope of the workshop to include systems like those mentioned above. The workshop will enable participants to familiarize themselves with recent developments on new techniques and tools, novel applications, draw lessons from users' experiences and identify research problems and common areas relevant to the practice of Erlang, Erlang-like languages, functional programming, distribution, concurrency etc. We invite three types of submissions. 1. Technical papers describing language extensions, critical discussions of the status quo, formal semantics of language constructs, program analysis and transformation, virtual machine extensions and compilation techniques, implementations and interfaces of Erlang in/with other languages, and new tools (profilers, tracers, debuggers, testing frameworks, etc.). Submission related to Erlang, Elixir, Scala/Akka, CloudHaskell, Lisp Flavoured Erlang, OCaml, and functional programming are welcome and encouraged. The maximum length for technical papers is restricted to 12 pages for full papers, and 6 pages for short papers. 2. Practice and application papers describing uses of Erlang in the "real-world", Erlang libraries for specific tasks, experiences from using Erlang in specific application domains, reusable programming idioms and elegant new ways of using Erlang to approach or solve a particular problem. The maximum length for the practice and application papers is restricted to 12 pages for full papers, and 6 pages for short papers. 3. Poster presentations describing topics related to the workshop goals. Each includes a maximum of 2 pages of the abstract and summary. Presentations in this category will be given an hour of shared simultaneous demonstration time. Note that the above are maximum lengths; we welcome shorter papers also, and the program committee will evaluate all papers on an equal basis independent of their lengths. Workshop Co-Chairs ------------------ Natalia Chechina, Bournemouth University, UK Adrian Francalanza, University of Malta, Malta Program Committee ----------------------------- (Note: the Workshop Co-Chairs are also committee members) Scott Lystig Fritchie, Wallaroo, USA James S. Larson, Google, USA Richard Carlsson, Klarna, Sweden Laura M. Castro, University of A Coru?a Vikt?ria F?rd?s, Klarna, Sweden Laura Bocchi, University of Kent, UK Rumyana Neykova, Imperial College, UK Roland Kuhn, Actyx, Germany Rocco De Nicola, IMT Lucca, Italy Emanuele D?Osulado, Imperial College, UK Mark Sheldon, Tufts Univesrity, USA James Fish, Pinterest, USA Tam?s Kozsik, E?tv?s Lor?nd University, Hungary Important Dates ----------------------- Submission deadline: Fri June 15, 2018 Author notification: Fri July 13, 2018 Final submission for the publisher: Sat August 4, 2018 Workshop date: Sat September 29, 2018 Instructions to authors -------------------------------- Papers must be submitted online via EasyChair (via the "Erlang2018" event). The submission page is https://easychair.org/conferences/?conf=erlang18. Submitted papers should be in portable document format (PDF), formatted using the ACM SIGPLAN style guidelines http://sigplan.org/Resources/Author/#acmart-format Each submission must adhere to SIGPLAN's republication policy. Violation risks summary rejection of the offending submission. Accepted papers will be published by the ACM and will appear in the ACM Digital Library. AUTHORS TAKE NOTE: The official publication date is the date the proceedings are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of your conference. The official publication date affects the deadline for any patent filings related to published work. The official publication date is the date the papers are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work. Paper submissions will be considered for poster submission in the case they are not accepted as full papers. Venue & Registration Details ------------------------------------------ For registration, please see the ICFP 2018 web site at: http://icfp18.sigplan.org/ Related Links -------------------- ICFP 2018 web site: http://icfp18.sigplan.org/ Past ACM SIGPLAN Erlang workshops: http://www.erlang.org/workshop/ Open Source Erlang: http://www.erlang.org/ EasyChair submission site: https://easychair.org/conferences/?conf=erlang18 ACM template: http://sigplan.org/Resources/Author/#acmart-format Author Information for SIGPLAN Conferences: https://www.acm.org/publications/gi-proceedings Atendee Information for SIGPLAN Events: http://www.sigplan.org/Resources/Policies/CodeOfConduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Mon Jun 11 17:15:43 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 11 Jun 2018 17:15:43 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: References: <20180611084533.GA61770@erix.ericsson.se> Message-ID: <20180611151543.GA85921@erix.ericsson.se> On Mon, Jun 11, 2018 at 01:07:31PM +0200, Frans Schneider wrote: > Thanks Raimo, > > See your point, but I am not yet really convinced. > Using a helper function would in my case.opinion make the code more > obliterative. I think the following code makes it very clear what goes > on while some helper function called from whatever other place would > make things much more obscure. > > handle_event(enter, > _Old_state, > #state{reliability_level = reliable, > state1 = announcing} = State, > #data{changes = []} = Data) -> > {next_state, State#state{state1 = idle}, Data}; > handle_event(enter, > _Old_state, > #state{reliability_level = reliable, > state1 = announcing}, > #data{push_mode = true} = Data -> > {next_state, State#state{state1 = pushing}, Data}; > handle_event(enter, > _Old_state, > #state{reliability_level = reliable, > state1 = announcing}, > #data{push_mode = false, > heartbeat_period = Heartbeat_period} = Data) -> > Hb_timer_ref = erlang:start_timer(Heartbeat_period, self(), hb), > {keep_state, Data#data{heartbeat_timer_ref = Hb_timer_ref}}; Why not use a generic timeout here? {keep_state_and_data, [{{timeout,heartbeat},Heartbeat_period,hb}]}; > > Currently, I resort to testing for a particular condition in the state > enter code and insert a next event (changes_not_empty) to trigger the > state change I need, which however also introduces extra code and makes > things also less obvious. > > Also, your co-locating argument is arguable since when using complex Yes, I might be biased towards state_functions, but in that mode it is a valid argument... And allowing it would be against the philosophy of that mode. Increasing freedom is not always a good thing. ;-( And allowing it for only one mode would be against the principle of least surprise. > states and handle_events, I tend to use a different ordering of the > functions, mostly focusing on the type of events to handle. (I know, > 'enter' isn't an event of course.) The feature you ask for probably makes more sense for handle_event_function. But there are still some hairy semantics to get right: * If changing states in state_enter call hops more than once - what should OldState be after the first state_enter call? I.e how much of a state change should changing states in the state_enter call be regarded as. As the gen_statem code looks now it seems to be easier to retain the original OldState and not regard the state change as complete until the final state of the state change has been deduced. * If changing states back to the OldState - should postponed events be retried? I think not - you in effect stayed in the same state. * Same again: if changing states back to the OldState - should the state enter call of the OldState be re-run or not? As the gen_statem code looks now it seems to be hard to see this as anything else than staying in the same state hence not running the state_enter call. Is that intuitive? * Should we also allow {next_event,...} from a state_enter call? * There might be more... I think the semantics is hairy enough already, so I still have a bad feeling about this... But I have asked some colleauages to also contemplate it, we'll see if they can find the time since we are in the middle of the 21.0 release circus. The only thing that _has_ to be prohibited from a state_enter call is 'postpone' since there is no event to postpone. The only reason I have for prohibiting state change and event insertion from state_enter calls is that I am afraid they would get too hairy semantics and hence cause too hard to understand state machines. / Raimo > > Frans > > > On 06/11/2018 10:45 AM, Raimo Niskanen wrote: > > On Mon, Jun 11, 2018 at 09:43:02AM +0200, Frans Schneider wrote: > >> Dear list, > >> > >> the state_enter_result(State) has as one its return values {next_state, > >> State, NewData, ...} but explicitly disallows the State to be different > >> from the current State,? First, I find it confusing to allow > >> 'next_state' here since State cannot change but the name makes the > > > > Throughout gen_statem is is by design so that there is no difference > > between {next_state, SameState, NewData, ...} > > and {keep_state, NewData, ...}. It is {keep_state, ...} that is shorthand > > for {next_state, SameState, ...}. > > > > So dissalowing {next_state, ...} in a state_enter call would be an > > exception to the main rule that {next_state, ...} is the generic form. > > > > There might be situations where you have a helper function that calculates > > the next state so it would want to always return {next_state, ...}, and > > then you want to use it in a state_enter call context knowing it will > > always return {next_state, SameState, ...}. I think this use should be > > possible. > > > >> suggestion it can. Secondly, I would love to be able to actually make a > >> state change. Quite often i find myself running a few tests on entering > >> a state and making state changes based on these tests. Now I have to > > > > The state_enter call is intended for code to be executed when entering a > > state, making this code co-located in the source with the state's event > > handling code. > > > > If the state_enter call should be allowed to change states it would no > > longer be co-located since it would for example start a timer for the other > > state it changes to. > > > > The alternative to using state_enter calls would be to have a convention > > where you at exit from the other state go through a helper function > > that does what you need for state entry, and remember to use this > > function for every state entry to the state in question. > > > > I think your use case is a good example for when having a specialized state > > change function makes sense. There is no single state this kind of code > > clearly should be co-located with. > > > >> move these tests to all states which can have this state as a result and > >> go directly to the end state. > >> > >> Would it be possible to change state from a state enter call? > > > > I do not like that idea, for the reasons above... > > > >> > >> Frans > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From fchschneider@REDACTED Mon Jun 11 17:47:30 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Mon, 11 Jun 2018 17:47:30 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180611151543.GA85921@erix.ericsson.se> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> Message-ID: On 06/11/2018 05:15 PM, Raimo Niskanen wrote: > On Mon, Jun 11, 2018 at 01:07:31PM +0200, Frans Schneider wrote: >> Thanks Raimo, >> >> See your point, but I am not yet really convinced. >> Using a helper function would in my case.opinion make the code more >> obliterative. I think the following code makes it very clear what goes >> on while some helper function called from whatever other place would >> make things much more obscure. >> >> handle_event(enter, >> _Old_state, >> #state{reliability_level = reliable, >> state1 = announcing} = State, >> #data{changes = []} = Data) -> >> {next_state, State#state{state1 = idle}, Data}; >> handle_event(enter, >> _Old_state, >> #state{reliability_level = reliable, >> state1 = announcing}, >> #data{push_mode = true} = Data -> >> {next_state, State#state{state1 = pushing}, Data}; >> handle_event(enter, >> _Old_state, >> #state{reliability_level = reliable, >> state1 = announcing}, >> #data{push_mode = false, >> heartbeat_period = Heartbeat_period} = Data) -> >> Hb_timer_ref = erlang:start_timer(Heartbeat_period, self(), hb), >> {keep_state, Data#data{heartbeat_timer_ref = Hb_timer_ref}}; > > > > Why not use a generic timeout here? > {keep_state_and_data, [{{timeout,heartbeat},Heartbeat_period,hb}]}; > > I have to be able to cancel the timer on entering the pushing state, so I need the timer reference. > >> >> Currently, I resort to testing for a particular condition in the state >> enter code and insert a next event (changes_not_empty) to trigger the >> state change I need, which however also introduces extra code and makes >> things also less obvious. >> >> Also, your co-locating argument is arguable since when using complex > > Yes, I might be biased towards state_functions, but in that mode it is a > valid argument... And allowing it would be against the philosophy of > that mode. Increasing freedom is not always a good thing. ;-( > And allowing it for only one mode would be against the principle of > least surprise. > > >> states and handle_events, I tend to use a different ordering of the >> functions, mostly focusing on the type of events to handle. (I know, >> 'enter' isn't an event of course.) > > The feature you ask for probably makes more sense for > handle_event_function. > > But there are still some hairy semantics to get right: > * If changing states in state_enter call hops more than once - what should > OldState be after the first state_enter call? I.e how much of a state > change should changing states in the state_enter call be regarded as. > As the gen_statem code looks now it seems to be easier to retain > the original OldState and not regard the state change as complete until > the final state of the state change has been deduced. Yes, I would expect a cascade of state enter calls with OldState reflecting the states it goes through. > * If changing states back to the OldState - should postponed events be > retried? I think not - you in effect stayed in the same state. > * Same again: if changing states back to the OldState - should the state > enter call of the OldState be re-run or not? As the gen_statem code > looks now it seems to be hard to see this as anything else than staying > in the same state hence not running the state_enter call. > Is that intuitive? > * Should we also allow {next_event,...} from a state_enter call? > * There might be more... > > I think the semantics is hairy enough already, so I still have a bad > feeling about this... But I have asked some colleauages to also > contemplate it, we'll see if they can find the time since we are in > the middle of the 21.0 release circus. > Great! Should I send you a reminder sometime? > The only thing that _has_ to be prohibited from a state_enter call is > 'postpone' since there is no event to postpone. > > The only reason I have for prohibiting state change and event insertion > from state_enter calls is that I am afraid they would get too hairy semantics > and hence cause too hard to understand state machines. > > / Raimo > > >> >> Frans >> >> >> On 06/11/2018 10:45 AM, Raimo Niskanen wrote: >>> On Mon, Jun 11, 2018 at 09:43:02AM +0200, Frans Schneider wrote: >>>> Dear list, >>>> >>>> the state_enter_result(State) has as one its return values {next_state, >>>> State, NewData, ...} but explicitly disallows the State to be different >>>> from the current State,? First, I find it confusing to allow >>>> 'next_state' here since State cannot change but the name makes the >>> >>> Throughout gen_statem is is by design so that there is no difference >>> between {next_state, SameState, NewData, ...} >>> and {keep_state, NewData, ...}. It is {keep_state, ...} that is shorthand >>> for {next_state, SameState, ...}. >>> >>> So dissalowing {next_state, ...} in a state_enter call would be an >>> exception to the main rule that {next_state, ...} is the generic form. >>> >>> There might be situations where you have a helper function that calculates >>> the next state so it would want to always return {next_state, ...}, and >>> then you want to use it in a state_enter call context knowing it will >>> always return {next_state, SameState, ...}. I think this use should be >>> possible. >>> >>>> suggestion it can. Secondly, I would love to be able to actually make a >>>> state change. Quite often i find myself running a few tests on entering >>>> a state and making state changes based on these tests. Now I have to >>> >>> The state_enter call is intended for code to be executed when entering a >>> state, making this code co-located in the source with the state's event >>> handling code. >>> >>> If the state_enter call should be allowed to change states it would no >>> longer be co-located since it would for example start a timer for the other >>> state it changes to. >>> >>> The alternative to using state_enter calls would be to have a convention >>> where you at exit from the other state go through a helper function >>> that does what you need for state entry, and remember to use this >>> function for every state entry to the state in question. >>> >>> I think your use case is a good example for when having a specialized state >>> change function makes sense. There is no single state this kind of code >>> clearly should be co-located with. >>> >>>> move these tests to all states which can have this state as a result and >>>> go directly to the end state. >>>> >>>> Would it be possible to change state from a state enter call? >>> >>> I do not like that idea, for the reasons above... >>> >>>> >>>> Frans >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From vladdu55@REDACTED Mon Jun 11 18:46:04 2018 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Mon, 11 Jun 2018 18:46:04 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> Message-ID: Hi! On Mon, Jun 11, 2018 at 5:47 PM Frans Schneider wrote: > > > On 06/11/2018 05:15 PM, Raimo Niskanen wrote: > > But there are still some hairy semantics to get right: > > * If changing states in state_enter call hops more than once - what > should > > OldState be after the first state_enter call? I.e how much of a state > > change should changing states in the state_enter call be regarded as. > > As the gen_statem code looks now it seems to be easier to retain > > the original OldState and not regard the state change as complete > until > > the final state of the state change has been deduced. > > Yes, I would expect a cascade of state enter calls with OldState > reflecting the states it goes through. > I am not a state machine expert, so maybe I am misunderstanding. If we allow jumping to arbitrary states from state_enter, isn't it like there is another state embedded there? One of the advantages of state machines is that it's easy to see from the code where the continuations are. One can no longer tell from examining the source state what states follow it, but one has to examine the state_enter of all these first. Isn't it cleaner to just separate this complex state into several simpler ones? Or am I missing something? best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Jun 11 18:47:43 2018 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 11 Jun 2018 18:47:43 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <4C1BF0DD-3AE3-4F75-9A7D-9130F7509D41@mac.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <4C1BF0DD-3AE3-4F75-9A7D-9130F7509D41@mac.com> Message-ID: Hello, Great discussion and ideas here! One thing that I've not seen mentioned is; what if the list representation was made more memory efficient? Today its 16 bytes per codepoint vs binaries that are 1-4 byte per codepoint. What if lists only used 8 bytes for each codepoint? what if it used the same as binaries? How would that change this discussion? On Mon, Jun 11, 2018 at 10:22 AM, Sean Hinde wrote: > Would an EEP help the existing work of the OTP team in this area or is > there already a clear plan and this would be a distraction? > > There is no plan about what should be done in this area. We want to continue developing the possibility to encode and decode protocols. We've had numerous discussions about how we would like to extent the binary syntax (or the syntax in general) in order to make it better for both novice and advanced users of Erlang, but have yet to come up with something that we like. So far our discussions have been mostly about decoding protocols, because we see that as the larger pain point, but maybe we were wrong about that? Regarding creating a new text type, I'm personally skeptical, but haven't formed a strong opinion on the matter yet. Adding a new type is a huge undertaking and we should be very sure that what we want before doing it. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Jun 11 18:55:40 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 11 Jun 2018 12:55:40 -0400 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180611151543.GA85921@erix.ericsson.se> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> Message-ID: <20180611165539.GA8817@ferdmbp.local> On 06/11, Raimo Niskanen wrote: >The feature you ask for probably makes more sense for >handle_event_function. > >But there are still some hairy semantics to get right: >* If changing states in state_enter call hops more than once - what should > OldState be after the first state_enter call? I.e how much of a state > change should changing states in the state_enter call be regarded as. > As the gen_statem code looks now it seems to be easier to retain > the original OldState and not regard the state change as complete until > the final state of the state change has been deduced. >* If changing states back to the OldState - should postponed events be > retried? I think not - you in effect stayed in the same state. >* Same again: if changing states back to the OldState - should the state > enter call of the OldState be re-run or not? As the gen_statem code > looks now it seems to be hard to see this as anything else than staying > in the same state hence not running the state_enter call. > Is that intuitive? >* Should we also allow {next_event,...} from a state_enter call? >* There might be more... - Would the state enter represent having entered the state or entering the state as soon as the callback returns? If it represents having entered the state: - postponing would conceptually make sense, even if it won't in practice (because you may just forever get back to the postponing state) - trace/debug calls need to show the state transition as having happened with no actual events being generated (or rather event a caused state to change to A, and then state changes B and C happened with no prompting) -- otherwise you need to consider state enter events the same way they'd be with external events and that's even conceptually worse - state timeouts are not ambiguous - if the events entered the state, it does solve the 'OldState' problem If it does not represent having entered the state - postponing would not conceptually make sense, and won't happen - no hard questions about whether enter state events are special events or not - state timeouts should possibly not be reset since you have never left the original state until at the end of evaluating the 'enter' chain of events - this ends up having the possibility to have a repeated 'OldState' value, which currently clashes with the expectation that the first enter event of a FSM is a duplicated one. This would make any double-transition look like the first transition of the process's loop. It seems that either way, there is a conceptual confusion as soon as an enter state event is allowed to force a change to a new state. I'd rather see the possibility forbidden. As for allowing next_event in an enter event, I'm not a fan. If you really need an event to run as the first thing in an enter event, just run it in the enter event itself? Otherwise this lets you conceptually do the same thing as multiple enter-state events with a state transition it seems. > >I think the semantics is hairy enough already, so I still have a bad >feeling about this... But I have asked some colleauages to also >contemplate it, we'll see if they can find the time since we are in >the middle of the 21.0 release circus. > Yes. One of the main complaints of gen_statem is that it is a complex behaviour and people take a long while to get adapted. It could be argued that the separation between enter and normal events is already part of that complexity, but in my opinion, not putting these barriers in place actually increase the overall potential system complexity even more. I still think gen_statem should have been simpler than it is today, so I'll be vocal about preventing further complexity from being added to it. >The only thing that _has_ to be prohibited from a state_enter call is >'postpone' since there is no event to postpone. > >The only reason I have for prohibiting state change and event insertion >from state_enter calls is that I am afraid they would get too hairy semantics >and hence cause too hard to understand state machines. > Yes, that is an entirely valid concern. I think it is worth making the existing typespecs a tad more complex to prevent user code from being even worse than that. Currently the typespecs are too obtuse to be good docs on their own (while they're kind of okay for supervisors and gen_servers), but they represent an intrinsinct complexity warning that a tutorial would have to alert the user to anyway if they were to be simplified. Might as well keep the type complexity and just have the tutorial fix things right away. I'm opposed to adding more conceptual complexity to gen_statem. Regards, Fred. From raimo+erlang-questions@REDACTED Tue Jun 12 08:29:42 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 12 Jun 2018 08:29:42 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> Message-ID: <20180612062942.GA53065@erix.ericsson.se> On Mon, Jun 11, 2018 at 05:47:30PM +0200, Frans Schneider wrote: > > > On 06/11/2018 05:15 PM, Raimo Niskanen wrote: > > On Mon, Jun 11, 2018 at 01:07:31PM +0200, Frans Schneider wrote: > >> Thanks Raimo, > >> > >> See your point, but I am not yet really convinced. > >> Using a helper function would in my case.opinion make the code more > >> obliterative. I think the following code makes it very clear what goes > >> on while some helper function called from whatever other place would > >> make things much more obscure. > >> > >> handle_event(enter, > >> _Old_state, > >> #state{reliability_level = reliable, > >> state1 = announcing} = State, > >> #data{changes = []} = Data) -> > >> {next_state, State#state{state1 = idle}, Data}; > >> handle_event(enter, > >> _Old_state, > >> #state{reliability_level = reliable, > >> state1 = announcing}, > >> #data{push_mode = true} = Data -> > >> {next_state, State#state{state1 = pushing}, Data}; > >> handle_event(enter, > >> _Old_state, > >> #state{reliability_level = reliable, > >> state1 = announcing}, > >> #data{push_mode = false, > >> heartbeat_period = Heartbeat_period} = Data) -> > >> Hb_timer_ref = erlang:start_timer(Heartbeat_period, self(), hb), > >> {keep_state, Data#data{heartbeat_timer_ref = Hb_timer_ref}}; > > > > > > > > Why not use a generic timeout here? > > {keep_state_and_data, [{{timeout,heartbeat},Heartbeat_period,hb}]}; > > > > > > I have to be able to cancel the timer on entering the pushing state, so > I need the timer reference. Cancel - set it to infinity: [{timeout,heartbeat},infinity,hb}] > > > > >> > >> Currently, I resort to testing for a particular condition in the state > >> enter code and insert a next event (changes_not_empty) to trigger the > >> state change I need, which however also introduces extra code and makes > >> things also less obvious. > >> > >> Also, your co-locating argument is arguable since when using complex > > > > Yes, I might be biased towards state_functions, but in that mode it is a > > valid argument... And allowing it would be against the philosophy of > > that mode. Increasing freedom is not always a good thing. ;-( > > And allowing it for only one mode would be against the principle of > > least surprise. > > > > > >> states and handle_events, I tend to use a different ordering of the > >> functions, mostly focusing on the type of events to handle. (I know, > >> 'enter' isn't an event of course.) > > > > The feature you ask for probably makes more sense for > > handle_event_function. > > > > But there are still some hairy semantics to get right: > > * If changing states in state_enter call hops more than once - what should > > OldState be after the first state_enter call? I.e how much of a state > > change should changing states in the state_enter call be regarded as. > > As the gen_statem code looks now it seems to be easier to retain > > the original OldState and not regard the state change as complete until > > the final state of the state change has been deduced. > > Yes, I would expect a cascade of state enter calls with OldState > reflecting the states it goes through. ...which would be the hard one to implement... > > > * If changing states back to the OldState - should postponed events be > > retried? I think not - you in effect stayed in the same state. > > * Same again: if changing states back to the OldState - should the state > > enter call of the OldState be re-run or not? As the gen_statem code > > looks now it seems to be hard to see this as anything else than staying > > in the same state hence not running the state_enter call. > > Is that intuitive? > > * Should we also allow {next_event,...} from a state_enter call? > > * There might be more... > > > > I think the semantics is hairy enough already, so I still have a bad > > feeling about this... But I have asked some colleauages to also > > contemplate it, we'll see if they can find the time since we are in > > the middle of the 21.0 release circus. > > > > Great! Should I send you a reminder sometime? After the release of 21.0, if this discussion dies off now. > > > The only thing that _has_ to be prohibited from a state_enter call is > > 'postpone' since there is no event to postpone. > > > > The only reason I have for prohibiting state change and event insertion > > from state_enter calls is that I am afraid they would get too hairy semantics > > and hence cause too hard to understand state machines. > > > > / Raimo > > > > > >> > >> Frans > >> > >> > >> On 06/11/2018 10:45 AM, Raimo Niskanen wrote: > >>> On Mon, Jun 11, 2018 at 09:43:02AM +0200, Frans Schneider wrote: > >>>> Dear list, > >>>> > >>>> the state_enter_result(State) has as one its return values {next_state, > >>>> State, NewData, ...} but explicitly disallows the State to be different > >>>> from the current State,? First, I find it confusing to allow > >>>> 'next_state' here since State cannot change but the name makes the > >>> > >>> Throughout gen_statem is is by design so that there is no difference > >>> between {next_state, SameState, NewData, ...} > >>> and {keep_state, NewData, ...}. It is {keep_state, ...} that is shorthand > >>> for {next_state, SameState, ...}. > >>> > >>> So dissalowing {next_state, ...} in a state_enter call would be an > >>> exception to the main rule that {next_state, ...} is the generic form. > >>> > >>> There might be situations where you have a helper function that calculates > >>> the next state so it would want to always return {next_state, ...}, and > >>> then you want to use it in a state_enter call context knowing it will > >>> always return {next_state, SameState, ...}. I think this use should be > >>> possible. > >>> > >>>> suggestion it can. Secondly, I would love to be able to actually make a > >>>> state change. Quite often i find myself running a few tests on entering > >>>> a state and making state changes based on these tests. Now I have to > >>> > >>> The state_enter call is intended for code to be executed when entering a > >>> state, making this code co-located in the source with the state's event > >>> handling code. > >>> > >>> If the state_enter call should be allowed to change states it would no > >>> longer be co-located since it would for example start a timer for the other > >>> state it changes to. > >>> > >>> The alternative to using state_enter calls would be to have a convention > >>> where you at exit from the other state go through a helper function > >>> that does what you need for state entry, and remember to use this > >>> function for every state entry to the state in question. > >>> > >>> I think your use case is a good example for when having a specialized state > >>> change function makes sense. There is no single state this kind of code > >>> clearly should be co-located with. > >>> > >>>> move these tests to all states which can have this state as a result and > >>>> go directly to the end state. > >>>> > >>>> Would it be possible to change state from a state enter call? > >>> > >>> I do not like that idea, for the reasons above... > >>> > >>>> > >>>> Frans > >>> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From raimo+erlang-questions@REDACTED Tue Jun 12 10:06:28 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 12 Jun 2018 10:06:28 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180611165539.GA8817@ferdmbp.local> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> Message-ID: <20180612080628.GB53065@erix.ericsson.se> On Mon, Jun 11, 2018 at 12:55:40PM -0400, Fred Hebert wrote: > On 06/11, Raimo Niskanen wrote: > >The feature you ask for probably makes more sense for > >handle_event_function. > > > >But there are still some hairy semantics to get right: > >* If changing states in state_enter call hops more than once - what should > > OldState be after the first state_enter call? I.e how much of a state > > change should changing states in the state_enter call be regarded as. > > As the gen_statem code looks now it seems to be easier to retain > > the original OldState and not regard the state change as complete until > > the final state of the state change has been deduced. > >* If changing states back to the OldState - should postponed events be > > retried? I think not - you in effect stayed in the same state. > >* Same again: if changing states back to the OldState - should the state > > enter call of the OldState be re-run or not? As the gen_statem code > > looks now it seems to be hard to see this as anything else than staying > > in the same state hence not running the state_enter call. > > Is that intuitive? > >* Should we also allow {next_event,...} from a state_enter call? > >* There might be more... > > - Would the state enter represent having entered the state or entering > the state as soon as the callback returns? > If it represents having entered the state: > - postponing would conceptually make sense, even if it won't in > practice (because you may just forever get back to the postponing > state) > - trace/debug calls need to show the state transition as having > happened with no actual events being generated (or rather event a > caused state to change to A, and then state changes B and C > happened with no prompting) -- otherwise you need to consider > state enter events the same way they'd be with external events and > that's even conceptually worse > - state timeouts are not ambiguous > - if the events entered the state, it does solve the 'OldState' > problem > If it does not represent having entered the state > - postponing would not conceptually make sense, and won't happen > - no hard questions about whether enter state events are special > events or not > - state timeouts should possibly not be reset since you have never > left the original state until at the end of evaluating the 'enter' > chain of events > - this ends up having the possibility to have a repeated 'OldState' > value, which currently clashes with the expectation that the first > enter event of a FSM is a duplicated one. This would make any > double-transition look like the first transition of the process's > loop. Sorry - I do not follow your argument about postponing making sense for "state entered" but not for "entering state"... I once contemplated having 'enter' as a pure automatically inserted event then possible to postpone, insert, and so on, but realized the hairiness that state machines utilizing that would achieve. So I went with 'enter' is not an event - it is a special call, so I will have no discussion about postponing it! Therefore I want it to be called "state enter call", not anything implicating "event"... > > It seems that either way, there is a conceptual confusion as soon as an > enter state event is allowed to force a change to a new state. I'd > rather see the possibility forbidden. > > As for allowing next_event in an enter event, I'm not a fan. If you > really need an event to run as the first thing in an enter event, just > run it in the enter event itself? Otherwise this lets you conceptually You lost me there... > do the same thing as multiple enter-state events with a state transition > it seems. I decided to have the principal that activating or deactivating state enter calls should not affect event handling in any way. I.e state enter calls should be orthogonal to 'postpone' and {next_event, ...}. Therefore you are now forced to insert events from the previous state, and you get the inserted event(s) after the state enter call. > > > > > >I think the semantics is hairy enough already, so I still have a bad > >feeling about this... But I have asked some colleauages to also > >contemplate it, we'll see if they can find the time since we are in > >the middle of the 21.0 release circus. > > > > Yes. One of the main complaints of gen_statem is that it is a complex > behaviour and people take a long while to get adapted. It could be > argued that the separation between enter and normal events is already > part of that complexity, but in my opinion, not putting these barriers > in place actually increase the overall potential system complexity even > more. I agree. State enter calls were introduced to allow code to be run when a state is entered, without having to duplicate that code at every state exit from the previous state. It is a feature existing in other state machine description languages e.g OpenBSD's ifstated. The implementation now is for the interpretation of "state enter" as "having entered the state", which I think is the safest and least complicated interpretation. But the question about the limitations of state enter calls has come up some times now, so maybe it was time to have a second look... The limitations are geared towards state oriented code, in particular callback mode state_functions, so the question is if it would be possible to open up flexibility that would be useful for non state oriented code, without messing up? > > I still think gen_statem should have been simpler than it is today, so > I'll be vocal about preventing further complexity from being added to > it. > > >The only thing that _has_ to be prohibited from a state_enter call is > >'postpone' since there is no event to postpone. > > > >The only reason I have for prohibiting state change and event insertion > >from state_enter calls is that I am afraid they would get too hairy semantics > >and hence cause too hard to understand state machines. > > > > Yes, that is an entirely valid concern. I think it is worth making the > existing typespecs a tad more complex to prevent user code from being > even worse than that. Can you elaborate on how to improve the typespecs for this? > > Currently the typespecs are too obtuse to be good docs on their own > (while they're kind of okay for supervisors and gen_servers), but they > represent an intrinsinct complexity warning that a tutorial would have > to alert the user to anyway if they were to be simplified. Might as well > keep the type complexity and just have the tutorial fix things right > away. I guess you are talking about the Reference Manual that is more or less built around the type specs, in contrast to the User's Guide (gen_statem Behaviour decription). From 21.0-rc2: http://erlang.org/documentation/doc-10.0-rc2/lib/stdlib-3.5/doc/html/gen_statem.html http://erlang.org/documentation/doc-10.0-rc2/doc/design_principles/statem.html Improvement suggestions on both are welcome! > > I'm opposed to adding more conceptual complexity to gen_statem. I think we are on the same side this time. :-) > > Regards, > Fred. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From fchschneider@REDACTED Tue Jun 12 10:31:27 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Tue, 12 Jun 2018 10:31:27 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180612080628.GB53065@erix.ericsson.se> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> Message-ID: <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> On 06/12/2018 10:06 AM, Raimo Niskanen wrote: > > I agree. State enter calls were introduced to allow code to be run when a > state is entered, without having to duplicate that code at every state exit > from the previous state. It is a feature existing in other state machine > description languages e.g OpenBSD's ifstated. > This is the exact reason for bringing up the whole issue. And sometimes, that requires a next_state NewState. > Frans From ingela.andin@REDACTED Tue Jun 12 12:31:51 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Tue, 12 Jun 2018 12:31:51 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> Message-ID: Hi! Could it be so that you are trying to solve the problem in the wrong place? State enter actions are good for example starting a timer, but I think it is illogical to change the state. It sound to me like you could do so pattern matching, maybe in conjunction with some guards in the your state function clauses and if your conditions are meet maybe all that clause does is to change state. Regards Ingela Erlang/OTP team - Ericsson AB 2018-06-12 10:31 GMT+02:00 Frans Schneider : > > > On 06/12/2018 10:06 AM, Raimo Niskanen wrote: > > >> I agree. State enter calls were introduced to allow code to be run when a >> state is entered, without having to duplicate that code at every state >> exit >> from the previous state. It is a feature existing in other state machine >> description languages e.g OpenBSD's ifstated. >> >> This is the exact reason for bringing up the whole issue. And sometimes, > that requires a next_state NewState. > >> >> > Frans > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fchschneider@REDACTED Tue Jun 12 14:14:45 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Tue, 12 Jun 2018 14:14:45 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> Message-ID: On 06/12/2018 12:31 PM, Ingela Andin wrote: > Hi! > > Could it be so that you are trying to solve the problem in the wrong > place? State enter actions are good for example starting a timer, but I > think it is illogical to change the state. > It sound to me like you could do so pattern matching, maybe in > conjunction with some guards in the your state function clauses and if > your conditions are meet maybe > all that clause does is to change state. > > Regards Ingela Erlang/OTP team - Ericsson AB > > Of course I can, but in this particular case it would lead to simpler code. For example, I have the following state changes defined ([1] par. 8.4.12.2): __|_____________|_____________________________|____________________ T2|waiting |HEARTBEAT message is received|if (HB.FinalFlag == | | | NOT_SET) then | | | must_send_ack else if | | | (HB.LivelinessFlag == | | | NOT_SET) then | | | may_send_ack | | |else | | | waiting __|_____________|_____________________________|____________________ T3|may_send_ack |GuardCondition: |waiting | | WP::missing_changes() == | | | | __|_____________|_____________________________|____________________ T4|may_send_ack |GuardCondition: |must_send_ack | | WP::missing_changes() != | | | | __|_____________|_____________________________|____________________ T5|must_send_ack|after(R::heartbeatResponseDelay) waiting __|_____________|_____________________________|____________________ Transitions T3/T4 are just begging for a enter state with next_state. The alternative would be to put the guard conditions at the end of T2, which of course is not that bad but it messes up the line of thinking of the original protocol designers and its readers. Also, T3/T4 could just become something like: handle_event(enter, _Old_state, #state{state2 = may_send_ack} = State, #data{missing_changes = []} = Data) -> {next_state, waiting, Data}; handle_event(enter, _Old_state, #state{state2 = may_send_ack} = State, Data) -> ... {next_state, must_send_ack, Data}; which I think is very accurate. In the example, there is only T2 as the old state but there are other examples with more than one originating states. Frans [1] https://www.omg.org/spec/DDSI-RTPS/2.2/PDF > > 2018-06-12 10:31 GMT+02:00 Frans Schneider >: > > > > On 06/12/2018 10:06 AM, Raimo Niskanen wrote: > > > I agree.? State enter calls were introduced to allow code to be > run when a > state is entered, without having to duplicate that code at every > state exit > from the previous state.? It is a feature existing in other > state machine > description languages e.g OpenBSD's ifstated. > > This is the exact reason for bringing up the whole issue. And > sometimes, that requires a next_state NewState. > > > > Frans > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > From ingela.andin@REDACTED Tue Jun 12 15:46:17 2018 From: ingela.andin@REDACTED (Ingela Andin) Date: Tue, 12 Jun 2018 15:46:17 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> Message-ID: Hi! Then I think it sounds like you could have a help function to call when you leave the previous states to determine what should be the next state so that you make the "TM" right transition in the first place. Or what am I not understanding? Regards Ingela Erlang/OTP team Ericsson AB 2018-06-12 14:14 GMT+02:00 Frans Schneider : > > > On 06/12/2018 12:31 PM, Ingela Andin wrote: > >> Hi! >> >> Could it be so that you are trying to solve the problem in the wrong >> place? State enter actions are good for example starting a timer, but I >> think it is illogical to change the state. >> It sound to me like you could do so pattern matching, maybe in >> conjunction with some guards in the your state function clauses and if your >> conditions are meet maybe >> all that clause does is to change state. >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> > Of course I can, but in this particular case it would lead to simpler > code. For example, I have the following state changes defined ([1] par. > 8.4.12.2): > > __|_____________|_____________________________|____________________ > T2|waiting |HEARTBEAT message is received|if (HB.FinalFlag == > | | | NOT_SET) then > | | | must_send_ack else if > | | | (HB.LivelinessFlag == > | | | NOT_SET) then > | | | may_send_ack > | | |else > | | | waiting > __|_____________|_____________________________|____________________ > T3|may_send_ack |GuardCondition: |waiting > | | WP::missing_changes() == | > | | | > __|_____________|_____________________________|____________________ > T4|may_send_ack |GuardCondition: |must_send_ack > | | WP::missing_changes() != | > | | | > __|_____________|_____________________________|____________________ > T5|must_send_ack|after(R::heartbeatResponseDelay) waiting > __|_____________|_____________________________|____________________ > > Transitions T3/T4 are just begging for a enter state with next_state. The > alternative would be to put the guard conditions at the end of T2, which of > course is not that bad but it messes up the line of thinking of the > original protocol designers and its readers. Also, T3/T4 could just become > something like: > > handle_event(enter, > _Old_state, > #state{state2 = may_send_ack} = State, > #data{missing_changes = []} = Data) -> > {next_state, waiting, Data}; > handle_event(enter, > _Old_state, > #state{state2 = may_send_ack} = State, > Data) -> > ... > {next_state, must_send_ack, Data}; > > which I think is very accurate. In the example, there is only T2 as the > old state but there are other examples with more than one originating > states. > > Frans > > [1] https://www.omg.org/spec/DDSI-RTPS/2.2/PDF > > >> 2018-06-12 10:31 GMT+02:00 Frans Schneider > >: >> >> >> >> On 06/12/2018 10:06 AM, Raimo Niskanen wrote: >> >> >> I agree. State enter calls were introduced to allow code to be >> run when a >> state is entered, without having to duplicate that code at every >> state exit >> from the previous state. It is a feature existing in other >> state machine >> description languages e.g OpenBSD's ifstated. >> >> This is the exact reason for bringing up the whole issue. And >> sometimes, that requires a next_state NewState. >> >> >> >> Frans >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Tue Jun 12 15:47:58 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 12 Jun 2018 15:47:58 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> Message-ID: <20180612134758.GA84130@erix.ericsson.se> On Tue, Jun 12, 2018 at 02:14:45PM +0200, Frans Schneider wrote: > > > On 06/12/2018 12:31 PM, Ingela Andin wrote: > > Hi! > > > > Could it be so that you are trying to solve the problem in the wrong > > place? State enter actions are good for example starting a timer, but I > > think it is illogical to change the state. > > It sound to me like you could do so pattern matching, maybe in > > conjunction with some guards in the your state function clauses and if > > your conditions are meet maybe > > all that clause does is to change state. > > > > Regards Ingela Erlang/OTP team - Ericsson AB > > > > > > Of course I can, but in this particular case it would lead to simpler > code. For example, I have the following state changes defined ([1] par. > 8.4.12.2): > > __|_____________|_____________________________|____________________ > T2|waiting |HEARTBEAT message is received|if (HB.FinalFlag == > | | | NOT_SET) then > | | | must_send_ack else if > | | | (HB.LivelinessFlag == > | | | NOT_SET) then > | | | may_send_ack > | | |else > | | | waiting > __|_____________|_____________________________|____________________ > T3|may_send_ack |GuardCondition: |waiting > | | WP::missing_changes() == | > | | | > __|_____________|_____________________________|____________________ > T4|may_send_ack |GuardCondition: |must_send_ack > | | WP::missing_changes() != | > | | | > __|_____________|_____________________________|____________________ > T5|must_send_ack|after(R::heartbeatResponseDelay) waiting > __|_____________|_____________________________|____________________ > > Transitions T3/T4 are just begging for a enter state with next_state. > The alternative would be to put the guard conditions at the end of T2, > which of course is not that bad but it messes up the line of thinking of > the original protocol designers and its readers. Also, T3/T4 could just > become something like: > > handle_event(enter, > _Old_state, > #state{state2 = may_send_ack} = State, > #data{missing_changes = []} = Data) -> > {next_state, waiting, Data}; > handle_event(enter, > _Old_state, > #state{state2 = may_send_ack} = State, > Data) -> > ... > {next_state, must_send_ack, Data}; > > which I think is very accurate. In the example, there is only T2 as the > old state but there are other examples with more than one originating > states. > > Frans > > [1] https://www.omg.org/spec/DDSI-RTPS/2.2/PDF Your example illustrates the problem. But I think the specification in question is very strange, and an ill fit for an event driven state machine. I can see that "HEARTBEAT message is received" is an event, but think it is very strange that [WP::missing_changes() == ] is an event - it seems more like a condition on the state, and hence the state may_send_ack appears to not be a state but instead some kind of decision point. So the state chart "8.24 - Behavior of the Reliable StatefulReader with respect to each matched Writer" seems to be a mix of a state diagram and a flowchart. Also, specifying a state machine as a table of transitions is a bit odd, but gets even more peculiar when the events are e.g "RTPS Reader is configured with a matched RTPS Writer." or "GuardCondition: WP::missing_changes() == " - here again a mix between a state transition table and flowchart conditions. Since the specification in question seems to be quite far from an event driven state machine it is no wonder if it seems to fit gen_statem poorly... :-( Implementing flowchart logic via pseudo state changes calling back and forth between the behaviour engine and the callback module is strange and inefficient, but in this case has the advantage of making the implementation kind of match the specification. So the question boils down to; would allowing state change from state enter calls be bad for what gen_statem is intended to be good at? / Raimo > > > > > 2018-06-12 10:31 GMT+02:00 Frans Schneider > >: > > > > > > > > On 06/12/2018 10:06 AM, Raimo Niskanen wrote: > > > > > > I agree.? State enter calls were introduced to allow code to be > > run when a > > state is entered, without having to duplicate that code at every > > state exit > > from the previous state.? It is a feature existing in other > > state machine > > description languages e.g OpenBSD's ifstated. > > > > This is the exact reason for bringing up the whole issue. And > > sometimes, that requires a next_state NewState. > > > > > > > > Frans > > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From john.hogberg@REDACTED Tue Jun 12 16:17:52 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Tue, 12 Jun 2018 14:17:52 +0000 Subject: [erlang-questions] Maybe potential incompatibility of file_info record In-Reply-To: References: Message-ID: <1528813071.13382.114.camel@ericsson.com> Hi, This is an intentional bugfix, and I've added a release note for it now. Thanks for reporting it! Regards, John H?gberg On tor, 2018-06-07 at 17:38 +0900, Shunichi Shinohara wrote: > Thanks a lot!! > > 2018-06-07 17:35 GMT+09:00 Bj?rn Gustavsson : > > > > Thanks for digging deeper. > > > > The old efile_driver has been rewritten and > > replaced with a NIF. I am not sure whether > > this change is an oversight/bug or a deliberate > > change. We will look into it before the release > > of OTP 21 and either fix the bug or add a > > note about the potential incompatibility. > > > > /Bj?rn > > > > > > > > On Thu, Jun 7, 2018 at 8:52 AM, Shunichi Shinohara > l.com> wrote: > > > > > > I looked into the source code and I think the difference between > > > 20 > > > and 21.0-rc2 is identified. > > > > > > # The subject of this thread is not suitable, > > > # but please let me report to not leave just a wrong information. > > > > > > First, the difference between OTP 20.3.4 and 21.0-rc2 is as > > > follows. > > > (on macOS, OTPs are installed from git by kerl) > > > > > > Setup file a and b, a is owned by root user. > > > % rm -f a && rm -f b && sudo touch a && sudo chmod 755 a && touch > > > b > > > > > > Let's start OTP 21.0-rc2 and read file_info of a and write it to > > > b, > > > it fails: > > > % ~/local/otp/OTP-21.0-rc2/bin/erl > > > Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-10.0] [source] [64-bit] > > > [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [sharing- > > > preserving] > > > > > > Eshell V10.0??(abort with ^G) > > > 1> {ok, Fi} = file:read_file_info("a"). > > > {ok,#file_info{size = 0,type = regular,access = read, > > > ???????????????atime = {{2018,6,7},{15,32,41}}, > > > ???????????????mtime = {{2018,6,7},{15,32,41}}, > > > ???????????????ctime = {{2018,6,7},{15,32,41}}, > > > ???????????????mode = 33261,links = 1,major_device = 16777220, > > > ???????????????minor_device = 0,inode = 17846256,uid = 0,gid = > > > 20}} > > > 2> file:write_file_info("b", Fi). > > > {error,eperm} > > > > > > Then, start OTP 20.3.4 and replay the same step, it succeeds: > > > % ~/local/otp/OTP-20.3.4/bin/erl > > > Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] > > > [async-threads:10] [hipe] [kernel-poll:false] > > > > > > Eshell V9.3??(abort with ^G) > > > 1> {ok, F} = file:read_file_info("a"). > > > {ok,#file_info{size = 0,type = regular,access = read, > > > ???????????????atime = {{2018,6,7},{15,32,41}}, > > > ???????????????mtime = {{2018,6,7},{15,32,41}}, > > > ???????????????ctime = {{2018,6,7},{15,32,41}}, > > > ???????????????mode = 33261,links = 1,major_device = 16777220, > > > ???????????????minor_device = 0,inode = 17846256,uid = 0,gid = > > > 20}} > > > 2> file:write_file_info("b", F). > > > ok > > > > > > The file_info records are the same (sorry again...), but the > > > results differ. > > > > > > Looking into unix_efile.c on maint-20 branch: > > > ????https://github.com/erlang/otp/blob/maint-20/erts/emulator/dri > > > vers/unix/unix_efile.c#L572 > > > it ignores eperm error and proceeds forward: > > > ????if (chown(name, pInfo->uid, pInfo->gid) && errno != EPERM) { > > > > > > I'm not sure the difference is judged as incompatibility or not, > > > it's nice if this note helps someone (including future me). > > > > > > Thanks, > > > Shino > > > > > > 2018-06-07 15:07 GMT+09:00 Shunichi Shinohara > > om>: > > > > > > > > I'm sorry I was wrong. Thanks for correct me, Bjorn. > > > > I have to dig the root cause down ;) > > > > > > > > Shino > > > > > > > > 2018-06-07 15:04 GMT+09:00 Bj?rn Gustavsson : > > > > > > > > > > On Thu, Jun 7, 2018 at 7:39 AM, Shunichi Shinohara > > > > n@REDACTED> wrote: > > > > > [...] > > > > > > > > > > > > It seems that some fields are added to file_info record > > > > > > between 20 and 21-rc2, > > > > > "Seems"? > > > > > > > > > > No fields have been added for a very long time. The only > > > > > modifications done in recent releases are updates > > > > > to the types of the existing fields. The last such change > > > > > was made at the end of 2015. > > > > > > > > > > /Bjorn > > > > > > > > > > -- > > > > > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > > > > > > -- > > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From fchschneider@REDACTED Tue Jun 12 16:37:51 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Tue, 12 Jun 2018 16:37:51 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180612134758.GA84130@erix.ericsson.se> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> <20180612134758.GA84130@erix.ericsson.se> Message-ID: <2aeb241f-114f-a753-4547-f53ec29dd821@gmail.com> Indeed, the specs are a little weird. It's for a reason the organization calls itself OMG. Frans On 06/12/2018 03:47 PM, Raimo Niskanen wrote: > On Tue, Jun 12, 2018 at 02:14:45PM +0200, Frans Schneider wrote: >> >> >> On 06/12/2018 12:31 PM, Ingela Andin wrote: >>> Hi! >>> >>> Could it be so that you are trying to solve the problem in the wrong >>> place? State enter actions are good for example starting a timer, but I >>> think it is illogical to change the state. >>> It sound to me like you could do so pattern matching, maybe in >>> conjunction with some guards in the your state function clauses and if >>> your conditions are meet maybe >>> all that clause does is to change state. >>> >>> Regards Ingela Erlang/OTP team - Ericsson AB >>> >>> >> >> Of course I can, but in this particular case it would lead to simpler >> code. For example, I have the following state changes defined ([1] par. >> 8.4.12.2): >> >> __|_____________|_____________________________|____________________ >> T2|waiting |HEARTBEAT message is received|if (HB.FinalFlag == >> | | | NOT_SET) then >> | | | must_send_ack else if >> | | | (HB.LivelinessFlag == >> | | | NOT_SET) then >> | | | may_send_ack >> | | |else >> | | | waiting >> __|_____________|_____________________________|____________________ >> T3|may_send_ack |GuardCondition: |waiting >> | | WP::missing_changes() == | >> | | | >> __|_____________|_____________________________|____________________ >> T4|may_send_ack |GuardCondition: |must_send_ack >> | | WP::missing_changes() != | >> | | | >> __|_____________|_____________________________|____________________ >> T5|must_send_ack|after(R::heartbeatResponseDelay) waiting >> __|_____________|_____________________________|____________________ >> >> Transitions T3/T4 are just begging for a enter state with next_state. >> The alternative would be to put the guard conditions at the end of T2, >> which of course is not that bad but it messes up the line of thinking of >> the original protocol designers and its readers. Also, T3/T4 could just >> become something like: >> >> handle_event(enter, >> _Old_state, >> #state{state2 = may_send_ack} = State, >> #data{missing_changes = []} = Data) -> >> {next_state, waiting, Data}; >> handle_event(enter, >> _Old_state, >> #state{state2 = may_send_ack} = State, >> Data) -> >> ... >> {next_state, must_send_ack, Data}; >> >> which I think is very accurate. In the example, there is only T2 as the >> old state but there are other examples with more than one originating >> states. >> >> Frans >> >> [1] https://www.omg.org/spec/DDSI-RTPS/2.2/PDF > > Your example illustrates the problem. > > But I think the specification in question is very strange, and an ill fit > for an event driven state machine. > > I can see that "HEARTBEAT message is received" is an event, but think it is > very strange that [WP::missing_changes() == ] is an event - it seems > more like a condition on the state, and hence the state may_send_ack appears > to not be a state but instead some kind of decision point. > > So the state chart "8.24 - Behavior of the Reliable StatefulReader with > respect to each matched Writer" seems to be a mix of a state diagram and a > flowchart. > > Also, specifying a state machine as a table of transitions is a bit odd, but > gets even more peculiar when the events are e.g "RTPS Reader is configured with > a matched RTPS Writer." or "GuardCondition: WP::missing_changes() == " > - here again a mix between a state transition table and flowchart conditions. > > Since the specification in question seems to be quite far from an event driven > state machine it is no wonder if it seems to fit gen_statem poorly... :-( > > Implementing flowchart logic via pseudo state changes calling back and > forth between the behaviour engine and the callback module is strange and > inefficient, but in this case has the advantage of making the > implementation kind of match the specification. > > So the question boils down to; would allowing state change from state enter > calls be bad for what gen_statem is intended to be good at? > > / Raimo > > > >> >>> >>> 2018-06-12 10:31 GMT+02:00 Frans Schneider >> >: >>> >>> >>> >>> On 06/12/2018 10:06 AM, Raimo Niskanen wrote: >>> >>> >>> I agree.? State enter calls were introduced to allow code to be >>> run when a >>> state is entered, without having to duplicate that code at every >>> state exit >>> from the previous state.? It is a feature existing in other >>> state machine >>> description languages e.g OpenBSD's ifstated. >>> >>> This is the exact reason for bringing up the whole issue. And >>> sometimes, that requires a next_state NewState. >>> >>> >>> >>> Frans >>> > From t@REDACTED Tue Jun 12 17:56:28 2018 From: t@REDACTED (Tristan Sloughter) Date: Tue, 12 Jun 2018 09:56:28 -0600 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <4C1BF0DD-3AE3-4F75-9A7D-9130F7509D41@mac.com> Message-ID: <1528818988.141168.1405522872.1F507DCD@webmail.messagingengine.com> That would be great! Would there be much reason at all to use binary for text if this were the case now that utf is also supported? I suppose it would still be optimal if one is passing around large chunks of >64 bytes of text, but besides that are there any performance reasons to use binaries over lists assuming the memory usage were the same? Tristan On Mon, Jun 11, 2018, at 10:47 AM, Lukas Larsson wrote: > Hello, > > Great discussion and ideas here! > > One thing that I've not seen mentioned is; what if the list > representation was made more memory efficient? Today its 16 bytes per > codepoint vs binaries that are 1-4 byte per codepoint. What if lists > only used 8 bytes for each codepoint? what if it used the same as > binaries? How would that change this discussion?> > On Mon, Jun 11, 2018 at 10:22 AM, Sean Hinde > wrote:>> >> Would an EEP help the existing work of the OTP team in this area or >> is there already a clear plan and this would be a distraction?>> > > There is no plan about what should be done in this area. We want to > continue developing the possibility to encode and decode protocols. > We've had numerous discussions about how we would like to extent the > binary syntax (or the syntax in general) in order to make it better > for both novice and advanced users of Erlang, but have yet to come up > with something that we like. So far our discussions have been mostly > about decoding protocols, because we see that as the larger pain > point, but maybe we were wrong about that?> > Regarding creating a new text type, I'm personally skeptical, but > haven't formed a strong opinion on the matter yet. Adding a new type > is a huge undertaking and we should be very sure that what we want > before doing it.> > Lukas > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From joe@REDACTED Tue Jun 12 20:47:37 2018 From: joe@REDACTED (joe mcguckin) Date: Tue, 12 Jun 2018 11:47:37 -0700 Subject: [erlang-questions] Erlang info Message-ID: <392A867C-5509-4815-9E8F-52DEC348C8A1@via.net> I recall one of the design wins Erlang had was a switch from Ericsson? Was this a voice pbx or a packet data switch? I assume Erlang controlled the control plane functions: If it was a voice switch, surely they must have used C (or similar) to perform the low-level hardware interfacing stuff like voice transcoding, digitization, etc How easy is it for a task to fire off a new thread on a different server? Thanks, Joe. From jesper.louis.andersen@REDACTED Tue Jun 12 21:16:48 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 12 Jun 2018 21:16:48 +0200 Subject: [erlang-questions] Erlang info In-Reply-To: <392A867C-5509-4815-9E8F-52DEC348C8A1@via.net> References: <392A867C-5509-4815-9E8F-52DEC348C8A1@via.net> Message-ID: On Tue, Jun 12, 2018 at 9:00 PM joe mcguckin wrote: > If it was a voice switch, surely they must have used C (or similar) to > perform the low-level hardware interfacing stuff like voice transcoding, > digitization, etc > > Chances are C is too slow for this operation as well. You usually need either FPGAs or ASICs and then a (thin) C driver layer to interface with the Erlang Control Plane. But I have relatively little knowledge of the actual Ericsson projects and their inner workings. > How easy is it for a task to fire off a new thread on a different server? > Starting a new process on another Erlang node is rather easy. Calling eg, spawn(Node, Fun) will start running Fun on the node Node (which is usually on a different machine). However, in most systems which need this, you tend to build higher-level solutions out of the underlying primitives depending on what kind of fault-guarantees you are looking for and what kind of problem you are trying to solve. Because working at the lowest level requires a lot of attention to the small details which you would like to solve once and for all. The reason we tend to say process and not thread is because threads usually implies there is a shared data space all threads can access with locks around data so shared. First, this model is going to break down in a distributed setting (Unless you have direct DMA over the network in a HPC-setting). Second, an Erlang process logically do not share data with each other and owns its own data. The current implementation is very explicit about this, giving each process its own heap, stack and garbage collection. But one could imagine sharing the heap among all processes: because the language is functional, there is no way one process can overwrite the data of another. The biggest secret is that data is copied between processes, and usually not passed by reference (except a few types of data). Since messages are often small and since this improves data locality (especially when you are distributed), the overhead of doing so tend to be smaller than what people think it is. This also suggests a model where you tend to move the computation to where the data is, rather than moving the data to the computation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.elsgaard@REDACTED Tue Jun 12 21:25:30 2018 From: thomas.elsgaard@REDACTED (Thomas Elsgaard) Date: Tue, 12 Jun 2018 21:25:30 +0200 Subject: [erlang-questions] Erlang info In-Reply-To: References: <392A867C-5509-4815-9E8F-52DEC348C8A1@via.net> Message-ID: You are most likely thinking about the AXD301, it was born as an ATM switch. It used an combination of C/C++, Java and Erlang At that time (long time ago) The AXD 301 control system, release 3.2, consisted of roughly: ? 1 million lines of Erlang source code ? 400,000 lines of C/C++ code ? 13,000 lines of Java code (excluding comments and whitespace) In addition, some 500,000 lines of C code reside on device processors. In the control system, about 100,000 lines of C code constitutes 3rd party software which is started, configured and supervised by corresponding (tiny) Erlang programs. The above numbers do not include the actual Erlang/OTP product, which contains 240,000 lines of Erlang, 460,000 lines of C/C++ and 15,000 lines of Java. You can find more details including the numbers above here: http://www.erlang.se/publications/Ulf_Wiger.pdf Thomas On Tue, 12 Jun 2018 at 21:17 Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > On Tue, Jun 12, 2018 at 9:00 PM joe mcguckin wrote: > >> If it was a voice switch, surely they must have used C (or similar) to >> perform the low-level hardware interfacing stuff like voice transcoding, >> digitization, etc >> >> > Chances are C is too slow for this operation as well. You usually need > either FPGAs or ASICs and then a (thin) C driver layer to interface with > the Erlang Control Plane. But I have relatively little knowledge of the > actual Ericsson projects and their inner workings. > > >> How easy is it for a task to fire off a new thread on a different server? >> > > Starting a new process on another Erlang node is rather easy. Calling eg, > spawn(Node, Fun) will start running Fun on the node Node (which is usually > on a different machine). > > However, in most systems which need this, you tend to build higher-level > solutions out of the underlying primitives depending on what kind of > fault-guarantees you are looking for and what kind of problem you are > trying to solve. Because working at the lowest level requires a lot of > attention to the small details which you would like to solve once and for > all. > > The reason we tend to say process and not thread is because threads > usually implies there is a shared data space all threads can access with > locks around data so shared. First, this model is going to break down in a > distributed setting (Unless you have direct DMA over the network in a > HPC-setting). Second, an Erlang process logically do not share data with > each other and owns its own data. The current implementation is very > explicit about this, giving each process its own heap, stack and garbage > collection. But one could imagine sharing the heap among all processes: > because the language is functional, there is no way one process can > overwrite the data of another. > > The biggest secret is that data is copied between processes, and usually > not passed by reference (except a few types of data). Since messages are > often small and since this improves data locality (especially when you are > distributed), the overhead of doing so tend to be smaller than what people > think it is. This also suggests a model where you tend to move the > computation to where the data is, rather than moving the data to the > computation. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Jun 13 02:55:58 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 12 Jun 2018 20:55:58 -0400 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180612080628.GB53065@erix.ericsson.se> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> Message-ID: <20180613005557.GB8817@ferdmbp.local> On 06/12, Raimo Niskanen wrote: >Sorry - I do not follow your argument about postponing making sense for >"state entered" but not for "entering state"... > > Assume I have a state machine with 3 states: ,--------------------v ready -> moving -> stopped-, ^-------------------------' If the enter event is seen as taking place after the state transition has taken place, then a transition of the form: ready -> stopped -> ready where the transition from 'stopped -> ready' takes place with the 'enter' event means that postponing should fire messages enqueued while in the 'ready' state even if you're back in the 'ready' state, since you went through a successful state transition to a state other than the current one. On the other hand, if the 'enter' event is seen as happening before a transition, then the 'enter' event when getting to the 'stopped' state would instead be interpreted like: ready ---x stopped | '---> ready Where the enter event _prevents_ switching to the stopped state. In that case, then it would make sense not to de-queue events postponed during the 'ready' state. This chain can be as long as you please though and you could imagine something like: a -> b -> c -> b -> c -> b -> a and if all those moves to 'c' or 'b' take place in 'enter' events, then that's an interesting edge case to handle. >> As for allowing next_event in an enter event, I'm not a fan. If you >> really need an event to run as the first thing in an enter event, just >> run it in the enter event itself? Otherwise this lets you conceptually > >You lost me there... > THinking back again, this is not a significant point. You can disregard it. >I decided to have the principal that activating or deactivating >state enter calls should not affect event handling in any way. >I.e state enter calls should be orthogonal to >'postpone' and {next_event, ...}. > >Therefore you are now forced to insert events from the previous state, >and you get the inserted event(s) after the state enter call. > I'm happy with the current state of things. > >I agree. State enter calls were introduced to allow code to be run when a >state is entered, without having to duplicate that code at every state exit >from the previous state. It is a feature existing in other state machine >description languages e.g OpenBSD's ifstated. > >The implementation now is for the interpretation of "state enter" as >"having entered the state", which I think is the safest and least >complicated interpretation. > Agreed. >But the question about the limitations of state enter calls has come up >some times now, so maybe it was time to have a second look... > >The limitations are geared towards state oriented code, in particular >callback mode state_functions, so the question is if it would be possible >to open up flexibility that would be useful for non state oriented code, >without messing up? > I think the limitations should be the same whether you work with callbacks or with handle_event(...). They're different types of events. I wouldn't expect to use {reply, ..., ...} actions to a cast or internal event (an easy one since they don't carry a pid), and I shouldn't expect to postpone or next_event action on an enter event. There's already plenty of context-sensitive information. >> >> Yes, that is an entirely valid concern. I think it is worth making the >> existing typespecs a tad more complex to prevent user code from being >> even worse than that. > >Can you elaborate on how to improve the typespecs for this? > I can't, not without repetition. Currently the biggest problem is that edoc has you jump around a bit to avoid repetition in the source code. For example, if I'm looking at handle_event, I have the following return signature: HandleEventResult HandleEventResult = event_handler_result(state()) Jumping to: event_handler_result(StateType) = {next_state, NextState :: StateType, NewData :: data()} | {next_state, NextState :: StateType, NewData :: data(), Actions :: [action()] | action()} | state_callback_result(action()) which has me jump to action(): action() = postpone | {postpone, Postpone :: postpone()} | {next_event, EventType :: event_type(), EventContent :: term()} | enter_action() which has me jump to enter_action(): enter_action() = hibernate | {hibernate, Hibernate :: hibernate()} | timeout_action() | reply_action() ... and so on. I basically need to manually click links and jump through at least 3-4 levels of type definitions to know what I am allowed to return or not. Note that I've also skipped over branching, as I have avoided expanding state_callback_result(action()), timeout_action(), and reply_action(). By comparison, if the repetition took place in the code rather than the doc (or if edoc could expand the type tree), then I could see a single signature for the thing. It totally makes sense for the types to be defined that way in code, but they don't make for good documentation. Regards, Fred. From mononcqc@REDACTED Wed Jun 13 03:03:43 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 12 Jun 2018 21:03:43 -0400 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> Message-ID: <20180613010343.GC8817@ferdmbp.local> On 06/12, Frans Schneider wrote: > >__|_____________|_____________________________|____________________ >T2|waiting |HEARTBEAT message is received|if (HB.FinalFlag == > | | | NOT_SET) then > | | | must_send_ack else if > | | | (HB.LivelinessFlag == > | | | NOT_SET) then > | | | may_send_ack > | | |else > | | | waiting >__|_____________|_____________________________|____________________ >T3|may_send_ack |GuardCondition: |waiting > | | WP::missing_changes() == | > | | | >__|_____________|_____________________________|____________________ >T4|may_send_ack |GuardCondition: |must_send_ack > | | WP::missing_changes() != | > | | | >__|_____________|_____________________________|____________________ >T5|must_send_ack|after(R::heartbeatResponseDelay) waiting >__|_____________|_____________________________|____________________ > >Transitions T3/T4 are just begging for a enter state with next_state. >The alternative would be to put the guard conditions at the end of T2, >which of course is not that bad but it messes up the line of thinking >of the original protocol designers and its readers. Also, T3/T4 could >just become something like: > >handle_event(enter, > _Old_state, > #state{state2 = may_send_ack} = State, > #data{missing_changes = []} = Data) -> > {next_state, waiting, Data}; >handle_event(enter, > _Old_state, > #state{state2 = may_send_ack} = State, > Data) -> > ... > {next_state, must_send_ack, Data}; > >which I think is very accurate. In the example, there is only T2 as >the old state but there are other examples with more than one >originating states. > Have you considered: handle_event(..., ..., t2, Data) -> ... {next_state, may_send_ack, NewData, [{next_event, internal, switch_states}]}; handle_event(internal, switch_states, maybe_send_ack, Data) -> case Data of #data{missing_changes=[]} -> {next_state, waiting, Data}; _ -> {next_state, must_send_ack, Data} end; ... Using next_event to force-enqueue the next event lets you do a switch through a state transition based entirely on its internal state. It does require any transition to 'may_send_ack' to set that value, though. From shino.shun@REDACTED Wed Jun 13 03:45:11 2018 From: shino.shun@REDACTED (Shunichi Shinohara) Date: Wed, 13 Jun 2018 10:45:11 +0900 Subject: [erlang-questions] Maybe potential incompatibility of file_info record In-Reply-To: <1528813071.13382.114.camel@ericsson.com> References: <1528813071.13382.114.camel@ericsson.com> Message-ID: Thanks for the information. Then, rebar3 (through erlware_commons) should be fixed for OTP-21. Regards, Shino 2018-06-12 23:17 GMT+09:00 John H?gberg : > Hi, > > This is an intentional bugfix, and I've added a release note for it > now. Thanks for reporting it! > > Regards, > John H?gberg > > On tor, 2018-06-07 at 17:38 +0900, Shunichi Shinohara wrote: >> Thanks a lot!! >> >> 2018-06-07 17:35 GMT+09:00 Bj?rn Gustavsson : >> > >> > Thanks for digging deeper. >> > >> > The old efile_driver has been rewritten and >> > replaced with a NIF. I am not sure whether >> > this change is an oversight/bug or a deliberate >> > change. We will look into it before the release >> > of OTP 21 and either fix the bug or add a >> > note about the potential incompatibility. >> > >> > /Bj?rn >> > >> > >> > >> > On Thu, Jun 7, 2018 at 8:52 AM, Shunichi Shinohara > > l.com> wrote: >> > > >> > > I looked into the source code and I think the difference between >> > > 20 >> > > and 21.0-rc2 is identified. >> > > >> > > # The subject of this thread is not suitable, >> > > # but please let me report to not leave just a wrong information. >> > > >> > > First, the difference between OTP 20.3.4 and 21.0-rc2 is as >> > > follows. >> > > (on macOS, OTPs are installed from git by kerl) >> > > >> > > Setup file a and b, a is owned by root user. >> > > % rm -f a && rm -f b && sudo touch a && sudo chmod 755 a && touch >> > > b >> > > >> > > Let's start OTP 21.0-rc2 and read file_info of a and write it to >> > > b, >> > > it fails: >> > > % ~/local/otp/OTP-21.0-rc2/bin/erl >> > > Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-10.0] [source] [64-bit] >> > > [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [sharing- >> > > preserving] >> > > >> > > Eshell V10.0 (abort with ^G) >> > > 1> {ok, Fi} = file:read_file_info("a"). >> > > {ok,#file_info{size = 0,type = regular,access = read, >> > > atime = {{2018,6,7},{15,32,41}}, >> > > mtime = {{2018,6,7},{15,32,41}}, >> > > ctime = {{2018,6,7},{15,32,41}}, >> > > mode = 33261,links = 1,major_device = 16777220, >> > > minor_device = 0,inode = 17846256,uid = 0,gid = >> > > 20}} >> > > 2> file:write_file_info("b", Fi). >> > > {error,eperm} >> > > >> > > Then, start OTP 20.3.4 and replay the same step, it succeeds: >> > > % ~/local/otp/OTP-20.3.4/bin/erl >> > > Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] >> > > [async-threads:10] [hipe] [kernel-poll:false] >> > > >> > > Eshell V9.3 (abort with ^G) >> > > 1> {ok, F} = file:read_file_info("a"). >> > > {ok,#file_info{size = 0,type = regular,access = read, >> > > atime = {{2018,6,7},{15,32,41}}, >> > > mtime = {{2018,6,7},{15,32,41}}, >> > > ctime = {{2018,6,7},{15,32,41}}, >> > > mode = 33261,links = 1,major_device = 16777220, >> > > minor_device = 0,inode = 17846256,uid = 0,gid = >> > > 20}} >> > > 2> file:write_file_info("b", F). >> > > ok >> > > >> > > The file_info records are the same (sorry again...), but the >> > > results differ. >> > > >> > > Looking into unix_efile.c on maint-20 branch: >> > > https://github.com/erlang/otp/blob/maint-20/erts/emulator/dri >> > > vers/unix/unix_efile.c#L572 >> > > it ignores eperm error and proceeds forward: >> > > if (chown(name, pInfo->uid, pInfo->gid) && errno != EPERM) { >> > > >> > > I'm not sure the difference is judged as incompatibility or not, >> > > it's nice if this note helps someone (including future me). >> > > >> > > Thanks, >> > > Shino >> > > >> > > 2018-06-07 15:07 GMT+09:00 Shunichi Shinohara > > > om>: >> > > > >> > > > I'm sorry I was wrong. Thanks for correct me, Bjorn. >> > > > I have to dig the root cause down ;) >> > > > >> > > > Shino >> > > > >> > > > 2018-06-07 15:04 GMT+09:00 Bj?rn Gustavsson : >> > > > > >> > > > > On Thu, Jun 7, 2018 at 7:39 AM, Shunichi Shinohara > > > > > n@REDACTED> wrote: >> > > > > [...] >> > > > > > >> > > > > > It seems that some fields are added to file_info record >> > > > > > between 20 and 21-rc2, >> > > > > "Seems"? >> > > > > >> > > > > No fields have been added for a very long time. The only >> > > > > modifications done in recent releases are updates >> > > > > to the types of the existing fields. The last such change >> > > > > was made at the end of 2015. >> > > > > >> > > > > /Bjorn >> > > > > >> > > > > -- >> > > > > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB >> > >> > >> > -- >> > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From joe@REDACTED Tue Jun 12 22:55:04 2018 From: joe@REDACTED (joe mcguckin) Date: Tue, 12 Jun 2018 13:55:04 -0700 Subject: [erlang-questions] How does Erland get maximum performance from a multi-core cpu? Message-ID: <1ECFE83A-8330-4BA8-9124-36BF9FF9E8F1@via.net> If erlang is one large unix process with hundreds or thousands of it's own processes internally, how does Erlang make maximum use of a multi-core cpu? Wouldn?t Erlang be scheduled and run on a single core ? Thanks, joe Joe McGuckin ViaNet Communications joe@REDACTED 650-207-0372 cell 650-213-1302 office 650-969-2124 fax From bob@REDACTED Wed Jun 13 06:50:54 2018 From: bob@REDACTED (Bob Ippolito) Date: Tue, 12 Jun 2018 21:50:54 -0700 Subject: [erlang-questions] How does Erland get maximum performance from a multi-core cpu? In-Reply-To: <1ECFE83A-8330-4BA8-9124-36BF9FF9E8F1@via.net> References: <1ECFE83A-8330-4BA8-9124-36BF9FF9E8F1@via.net> Message-ID: Threads allow a single process to utilize multiple cores. Erlang takes advantage of them to run its schedulers. On Tue, Jun 12, 2018 at 20:16 joe mcguckin wrote: > > If erlang is one large unix process with hundreds or thousands of it's own > processes internally, how does Erlang make maximum use of a multi-core cpu? > > Wouldn?t Erlang be scheduled and run on a single core ? > > Thanks, > > joe > > Joe McGuckin > ViaNet Communications > > joe@REDACTED > 650-207-0372 cell > 650-213-1302 office > 650-969-2124 fax > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vances@REDACTED Wed Jun 13 07:38:39 2018 From: vances@REDACTED (Vance Shipley) Date: Wed, 13 Jun 2018 11:08:39 +0530 Subject: [erlang-questions] How does Erland get maximum performance from a multi-core cpu? In-Reply-To: <1ECFE83A-8330-4BA8-9124-36BF9FF9E8F1@via.net> References: <1ECFE83A-8330-4BA8-9124-36BF9FF9E8F1@via.net> Message-ID: Joe, The answer is multiple schedulers. In the days before the SMP enabled Erlang emulator (VM) we would need to run multiple nodes (VM) to take advantage of multiple cores. Now the default mode for the VM is to run as many schedulers as there are cores available. This also highly configurable. Although Erlang's distribution allows processes to transparently communicate across nodes there is a great performance advantage in intra-node messaging as there is reference passing and it avoids serialization in the external term format. It's interesting though how everything goes in cycles. Five years ago I was keeping the core counts very high, and the node counts very low, to get great performance. Now with cloud native we run a node in a container and communicate with other (microservices) nodes using either distribution or external protocols (e.g. REST) so the overheads are again much higher. But the old way was vertical scaling and this is horizontal scaling, we pay a tax but go from dozens of cores to hundreds or thousands. On Wed, Jun 13, 2018, 08:46 joe mcguckin wrote: > > If erlang is one large unix process with hundreds or thousands of it's own > processes internally, how does Erlang make maximum use of a multi-core cpu? > > Wouldn?t Erlang be scheduled and run on a single core ? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stamarit@REDACTED Wed Jun 13 09:50:46 2018 From: stamarit@REDACTED (Salvador Tamarit) Date: Wed, 13 Jun 2018 09:50:46 +0200 Subject: [erlang-questions] WFLP 2018: Call for Papers Message-ID: ==================================== WFLP 2018: Call for Papers ==================================== 26th International Workshop on Functional and Logic Programming WFLP 2018 http://ppdp-lopstr-18.cs.uni-frankfurt.de/wflp18.html Frankfurt, Germany, September 6, 2018 (co-located with PPDP 2018 and LOPSTR 2018) The international Workshop on Functional and (constraint) Logic Programming (WFLP) aims at bringing together researchers, students, and practitioners interested in functional programming, logic programming, and their integration. WFLP has a reputation for being a lively and friendly forum, and it is open for presenting and discussing work in progress, technical contributions, experience reports, experiments, reviews, and system descriptions. The 26th International Workshop on Functional and (constraint) Logic Programming (WFLP 2018) will be held at the Goethe-University Frankfurt am Main, Germany. Previous WFLP editions were WFLP 2017 (W?rzburg, Germany), WFLP 2016 (Leipzig, Germany), WFLP 2014 (Wittenberg, Germany), WFLP 2013 (Kiel, Germany), WFLP 2012 (Nagoya, Japan), WFLP 2011 (Odense, Denmark), WFLP 2010 (Madrid, Spain), WFLP 2009 (Brasilia, Brazil), WFLP 2008 (Siena, Italy), WFLP 2007 (Paris, France), WFLP 2006 (Madrid, Spain), WCFLP 2005 (Tallinn, Estonia), WFLP 2004 (Aachen, Germany), WFLP 2003 (Valencia, Spain), WFLP 2002 (Grado, Italy), WFLP 2001 (Kiel, Germany), WFLP 2000 (Benicassim, Spain), WFLP'99 (Grenoble, France), WFLP'98 (Bad Honnef, Germany), WFLP'97 (Schwarzenberg, Germany), WFLP'96 (Marburg, Germany), WFLP'95 (Schwarzenberg, Germany), WFLP'94 (Schwarzenberg, Germany), WFLP'93 (Rattenberg, Germany), and WFLP'92 (Karlsruhe, Germany). WFLP 2018 will be co-located with PPDP 2018 (International Symposium on Principles and Practice of Declarative Programming) and LOPSTR 2018 (International Symposium on Logic-Based Program Synthesis and Transformation. The topics of interest cover all aspects of functional ang logic programming. They include (but are not limited to): * Functional programming * Logic programming * Constraint programming * Deductive databases, data mining * Extensions of declarative languages, objects * Multi-paradigm declarative programming * Foundations, semantics, non-monotonic reasoning, dynamics * Parallelism, concurrency * Program analysis, abstract interpretation * program and model manipulation * Program transformation, partial evaluation, meta-programming * Specification, verification * Debugging, testing * Knowledge representation, machine learning * Interaction of declarative programming with other formalisms * Implementation of declarative languages * Advanced programming environments and tools * Software techniques for declarative programming * Applications The primary focus is on new and original research results, but submissions describing innovative products, prototypes under development, application systems, or interesting experiments (e.g., benchmarks) are also encouraged. Survey papers that present some aspects of the above topics from a new perspective, and experience reports are also welcome. Papers must be written and presented in English. Work that already appeared in unpublished or informally published workshop proceedings may be submitted (please contact the PC chair in case of questions). Important Dates Abstract submission: June 27, 2018 Paper submission: July 2, 2018 Notification: July 27, 2018 Camera-ready (for electronic pre-proceedings): August 24, 2018 Workshop: September 6, 2018 Submission Guidelines Submission is via Easychair submission website for WFLP 2018: https://easychair.org/conferences/?conf=wflp2018 Authors are invited to submit papers in the following categories: + Regular research paper + Work-in-progress report + System description Regular research papers must describe original work, be written and presented in English, and must not substantially overlap with papers that have been formally published or that are simultaneously submitted to a journal, conference, or workshop with formal proceedings. They will be judged on the basis of significance, relevance, correctness, originality, and clarity. For work-in-progress reports and system descriptions, less formal rules apply, and presentation-only submissions (talk and discussion, but no paper in the formal proceedings) are possible. Please contact the PC chair with any questions. All submissions must be formatted in the Lecture Notes in Computer Science style. Submissions cannot exceed 15 pages including references but excluding well-marked appendices not intended for publication. Reviewers are not required to read the appendices, and thus papers should be intelligible without them. However, all submissions (especially work-in-progress reports and system descriptions) may be considerably shorter than 15 pages. Proceedings All papers accepted for presentation at the conference will be published in informal proceedings publicly available at the Computing Research Repository. According to the program committee reviews, submissions can be directly accepted for publication in the formal post-conference proceedings. The formal post-conference proceedings will be published in both electronic and paper formats by Springer in the Lecture Notes in Computer Science series. After the conference, all authors accepted only for presentation will be invited to revise and/or extend their submissions in the light of the feedback solicited at the conference. Then, after another round of reviewing, these revised papers may also be published in the formal proceedings. Therefore, all accepted papers will be published in open-access, and the authors can also decide to publish their work in the Springer LNCS formal proceedings. Program Committee + Slim Abdennadher, German University in Cairo, Egypt + Maria Alpuente, Universitat Polit?cnica de Val?ncia, Spain + Sergio Antoy, Portland State University, USA + Olaf Chitil, University of Kent, UK + Maria del Mar Gallardo, Universidad de M?laga, Spain + Michael Hanus, University of Kiel, Germany + Herbert Kuchen, University of M?nster, Germany + Kostis Sagonas, Uppsala University, Sweden + Tom Schrijvers, Katholieke Universiteit Leuven, Belgium + Sibylle Schwarz, HTWK Leipzig, Germany + Martina Seidl, Johannes Kepler University Linz, Austria + Dietmar Seipel, University of W?rzburg, Germany + Josep Silva, Universitat Polit?cnica de Val?ncia, Spain (Program Chair) + Salvador Tamarit, Universidad Polit?cnica de Madrid, Spain + Janis Voigtl?nder, University of Duisburg-Essen, Germany + Johannes Waldmann, HTWK Leipzig, Germany Organizing Committee David Sabel (General Chair), Computer Science Institute Goethe-University Frankfurt am Main, Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Wed Jun 13 10:19:35 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 13 Jun 2018 10:19:35 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180613005557.GB8817@ferdmbp.local> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <20180613005557.GB8817@ferdmbp.local> Message-ID: <20180613081935.GA66019@erix.ericsson.se> Top posting, for once, since I totally agree! / Raimo On Tue, Jun 12, 2018 at 08:55:58PM -0400, Fred Hebert wrote: > On 06/12, Raimo Niskanen wrote: > >Sorry - I do not follow your argument about postponing making sense for > >"state entered" but not for "entering state"... > > > > > > Assume I have a state machine with 3 states: > > ,--------------------v > ready -> moving -> stopped-, > ^-------------------------' > > If the enter event is seen as taking place after the state transition > has taken place, then a transition of the form: > > ready -> stopped -> ready > > where the transition from 'stopped -> ready' takes place with the > 'enter' event means that postponing should fire messages enqueued while > in the 'ready' state even if you're back in the 'ready' state, since you > went through a successful state transition to a state other than the > current one. > > On the other hand, if the 'enter' event is seen as happening before a > transition, then the 'enter' event when getting to the 'stopped' state > would instead be interpreted like: > > ready ---x stopped > | > '---> ready > > Where the enter event _prevents_ switching to the stopped state. In that > case, then it would make sense not to de-queue events postponed during > the 'ready' state. > > This chain can be as long as you please though and you could imagine > something like: > > a -> b -> c -> b -> c -> b -> a > > and if all those moves to 'c' or 'b' take place in 'enter' events, then > that's an interesting edge case to handle. > > >> As for allowing next_event in an enter event, I'm not a fan. If you > >> really need an event to run as the first thing in an enter event, just > >> run it in the enter event itself? Otherwise this lets you conceptually > > > >You lost me there... > > > > THinking back again, this is not a significant point. You can disregard > it. > > >I decided to have the principal that activating or deactivating > >state enter calls should not affect event handling in any way. > >I.e state enter calls should be orthogonal to > >'postpone' and {next_event, ...}. > > > >Therefore you are now forced to insert events from the previous state, > >and you get the inserted event(s) after the state enter call. > > > > I'm happy with the current state of things. > > > > >I agree. State enter calls were introduced to allow code to be run when a > >state is entered, without having to duplicate that code at every state exit > >from the previous state. It is a feature existing in other state machine > >description languages e.g OpenBSD's ifstated. > > > >The implementation now is for the interpretation of "state enter" as > >"having entered the state", which I think is the safest and least > >complicated interpretation. > > > > Agreed. > > >But the question about the limitations of state enter calls has come up > >some times now, so maybe it was time to have a second look... > > > >The limitations are geared towards state oriented code, in particular > >callback mode state_functions, so the question is if it would be possible > >to open up flexibility that would be useful for non state oriented code, > >without messing up? > > > > I think the limitations should be the same whether you work with > callbacks or with handle_event(...). They're different types of events. > I wouldn't expect to use {reply, ..., ...} actions to a cast or internal > event (an easy one since they don't carry a pid), and I shouldn't expect > to postpone or next_event action on an enter event. > > There's already plenty of context-sensitive information. > > >> > >> Yes, that is an entirely valid concern. I think it is worth making the > >> existing typespecs a tad more complex to prevent user code from being > >> even worse than that. > > > >Can you elaborate on how to improve the typespecs for this? > > > > I can't, not without repetition. Currently the biggest problem is that > edoc has you jump around a bit to avoid repetition in the source code. > For example, if I'm looking at handle_event, I have the following return > signature: > > HandleEventResult > HandleEventResult = event_handler_result(state()) > > Jumping to: > > event_handler_result(StateType) = > {next_state, NextState :: StateType, NewData :: data()} | > {next_state, > NextState :: StateType, > NewData :: data(), > Actions :: [action()] | action()} | > state_callback_result(action()) > > > which has me jump to action(): > > action() = > postpone | > {postpone, Postpone :: postpone()} | > {next_event, > EventType :: event_type(), > EventContent :: term()} | > enter_action() > > which has me jump to enter_action(): > > enter_action() = > hibernate | > {hibernate, Hibernate :: hibernate()} | > timeout_action() | > reply_action() > > ... and so on. > > I basically need to manually click links and jump through at least 3-4 > levels of type definitions to know what I am allowed to return or not. > Note that I've also skipped over branching, as I have avoided expanding > state_callback_result(action()), timeout_action(), and reply_action(). > > By comparison, if the repetition took place in the code rather than the > doc (or if edoc could expand the type tree), then I could see a single > signature for the thing. > > It totally makes sense for the types to be defined that way in code, but > they don't make for good documentation. > > Regards, > Fred. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From raimo+erlang-questions@REDACTED Wed Jun 13 10:21:00 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 13 Jun 2018 10:21:00 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <2aeb241f-114f-a753-4547-f53ec29dd821@gmail.com> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> <20180612134758.GA84130@erix.ericsson.se> <2aeb241f-114f-a753-4547-f53ec29dd821@gmail.com> Message-ID: <20180613082100.GB66019@erix.ericsson.se> On Tue, Jun 12, 2018 at 04:37:51PM +0200, Frans Schneider wrote: > Indeed, the specs are a little weird. It's for a reason the organization > calls itself OMG. What was it called? ROFLMAO! > > Frans > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From fchschneider@REDACTED Wed Jun 13 10:25:06 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Wed, 13 Jun 2018 10:25:06 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <20180613010343.GC8817@ferdmbp.local> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> <20180613010343.GC8817@ferdmbp.local> Message-ID: <54e8714b-5c76-accb-ffa1-90a802434320@gmail.com> Well, that was an enlightening discussion. Just ditched the initial implementation and started all over again. Conditions now are true conditions and events are real events. The number of states now is half the number they were, but the handle_event functions tend to be more complex. I could remove the whole state enter functionality. Thanks, Frans On 06/13/2018 03:03 AM, Fred Hebert wrote: > On 06/12, Frans Schneider wrote: >> >> __|_____________|_____________________________|____________________ >> T2|waiting????? |HEARTBEAT message is received|if (HB.FinalFlag == >> ?|???????????? |???????????????????????????? |??? NOT_SET) then >> ?|???????????? |???????????????????????????? |? must_send_ack else if >> ?|???????????? |???????????????????????????? |?? (HB.LivelinessFlag == >> ?|???????????? |???????????????????????????? |?? NOT_SET) then >> ?|???????????? |???????????????????????????? |? may_send_ack >> ?|???????????? |???????????????????????????? |else >> ?|???????????? |???????????????????????????? |? waiting >> __|_____________|_____________________________|____________________ >> T3|may_send_ack |GuardCondition:????????????? |waiting >> ?|???????????? |?? WP::missing_changes() ==? | >> ?|???????????? |?????? ?????????????? | >> __|_____________|_____________________________|____________________ >> T4|may_send_ack |GuardCondition:????????????? |must_send_ack >> ?|???????????? |?? WP::missing_changes() !=? | >> ?|???????????? |?????? ?????????????? | >> __|_____________|_____________________________|____________________ >> T5|must_send_ack|after(R::heartbeatResponseDelay) waiting >> __|_____________|_____________________________|____________________ >> >> Transitions T3/T4 are just begging for a enter state with next_state. >> The alternative would be to put the guard conditions at the end of T2, >> which of course is not that bad but it messes up the line of thinking >> of the original protocol designers and its readers. Also, T3/T4 could >> just become something like: >> >> handle_event(enter, >> ???????? _Old_state, >> ???????? #state{state2 = may_send_ack} = State, >> ???????? #data{missing_changes = []} = Data) -> >> ?? {next_state, waiting, Data}; >> handle_event(enter, >> ???????? _Old_state, >> ???????? #state{state2 = may_send_ack} = State, >> ???????? Data) -> >> ?? ... >> ?? {next_state, must_send_ack, Data}; >> >> which I think is very accurate. In the example, there is only T2 as >> the old state but there are other examples with more than one >> originating states. >> > > Have you considered: > > handle_event(..., ..., t2, Data) -> > ?? ... > ?? {next_state, may_send_ack, NewData, > ??? [{next_event, internal, switch_states}]}; > handle_event(internal, switch_states, maybe_send_ack, Data) -> > ?? case Data of > ?????? #data{missing_changes=[]} -> > ?????????? {next_state, waiting, Data}; > ?????? _ -> > ?????????? {next_state, must_send_ack, Data} > ?? end; > ... > > > Using next_event to force-enqueue the next event lets you do a switch > through a state transition based entirely on its internal state. It does > require any transition to 'may_send_ack' to set that value, though. > From lukas@REDACTED Wed Jun 13 10:35:10 2018 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 13 Jun 2018 10:35:10 +0200 Subject: [erlang-questions] Binary string literal syntax In-Reply-To: <1528818988.141168.1405522872.1F507DCD@webmail.messagingengine.com> References: <754B2D08-C3DB-48D1-8329-1069BDDCFDD4@mac.com> <4C1BF0DD-3AE3-4F75-9A7D-9130F7509D41@mac.com> <1528818988.141168.1405522872.1F507DCD@webmail.messagingengine.com> Message-ID: On Tue, Jun 12, 2018 at 5:56 PM Tristan Sloughter wrote: > That would be great! > > Would there be much reason at all to use binary for text if this were the > case now that utf is also supported? > It may still be better to use binaries as text if you don't do much processing on the Erlang side, since the list would contain char() entries, there is still en encoding cost to convert to utf-whatever that may want to be avoided. Also you will read the data from somewhere which is probably utf-something encoded, so at least when doing the initial parsing you will deal with binaries. I suppose it would still be optimal if one is passing around large chunks > of >64 bytes of text, but besides that are there any performance reasons to > use binaries over lists assuming the memory usage were the same? > I'm not sure.... One of the things that binaries are good at is matching out sub binaries, i.e. taking <<"foo">> out of <<"foobar">> without having to copy <<"foo">>. In order to do the same with lists a new syntax would have to be added, something like [SubList:24/list | T] and a lot of support in the run-time. Today there are 4 different types of binaries in the run-time, while only 1 list type. If we go down this route we'll end up with 3-4 list types as well, which of course adds to complexity in a lot of places. I'm fairly confident that we can get the cost of lists down to 8 bytes per cons cell, but it will require re-writing a lot of code all over the place. Getting it down even further will be even more difficult.... but not impossible I think. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Jun 13 11:46:04 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 13 Jun 2018 11:46:04 +0200 Subject: [erlang-questions] [ANN] Asciideck: Asciidoc for Erlang Message-ID: <55342ba7-7313-db7f-d991-710659db3812@ninenines.eu> Hello, In my spare time I wrote an Asciidoc parser and translator project: https://github.com/ninenines/asciideck As far as I can tell this is the first Asciidoc implementation on BEAM. Three strong points: * It's not Markdown; Asciidoc is better defined and more powerful * The parser gives you an AST that you can further manipulate * The code is very readable, this is the code for parsing section titles for example: https://github.com/ninenines/asciideck/blob/master/src/asciideck_block_parser.erl#L134 - let it crash! I've switched ninenines.eu to use it and it builds the docs of all my projects. It also builds man pages of the projects. More work remains to be done but it's in a very good shape already. I've written about it in more details at: https://ninenines.eu/articles/asciideck/ Cheers, -- Lo?c Hoguin https://ninenines.eu From torben.lehoff@REDACTED Wed Jun 13 11:51:31 2018 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Wed, 13 Jun 2018 11:51:31 +0200 Subject: [erlang-questions] missing line numbers when gen_statem crashes Message-ID: Hi, I have a crashing gen_statem, but to the best of my knowledge I cannot see from the stock error messages which line my gen_statem dies on. I have searched high and low in my console output, but this is the only thing that I could find: =CRASH REPORT==== 13-Jun-2018::11:17:28 === crasher: initial call: eee_sched_shard:init/1 pid: <0.3149.0> registered_name: [] exception error: {badrecord,trans_opts} in function gen_statem:loop_event_actions_list/10 (gen_statem.erl, line 1210) ancestors: [<0.3148.0>,<0.3114.0>,<0.3107.0>,eee_sched_streams_sup, eee_sched_task_sup,eee_sched_sup,<0.2867.0>] message_queue_len: 0 messages: [] links: [<0.3148.0>] dictionary: [] trap_exit: false status: running heap_size: 6772 stack_size: 27 reductions: 58869 neighbours: I'm about to turn on debug mode, but I don't think that that should be the answer to this kind of problem. Line numbers should come out by default. Cheers, Torben -- https://www.linkedin.com/in/torbenhoffmann/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Jun 13 11:55:37 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 13 Jun 2018 11:55:37 +0200 Subject: [erlang-questions] How does Erland get maximum performance from a multi-core cpu? In-Reply-To: References: <1ECFE83A-8330-4BA8-9124-36BF9FF9E8F1@via.net> Message-ID: Another important point to add: Erlang doesn't make maximal usage of the CPU cores it is given. The system usually values low latency operation over throughput, so it has to forgo some raw throughput in order to achieve this goal. To be precise, there has to be a check in each loop in the program to make sure preemption can happen of a process which has run for too long on a CPU core. This check is a function call, because loops are implemented as tail-calling functions. In turn, you cannot maximize the CPU doing productive work as these checks will cost something. The tradeoff is also made in e.g., Go, and they have recently added an option to preempt loops as well. However, people are somewhat reluctant of that option since it hurts their throughput. These tradeoff turned out to be valuable in the original setting of the Erlang language, but it also works for modern distributed systems. On Wed, Jun 13, 2018 at 7:39 AM Vance Shipley wrote: > Joe, > > The answer is multiple schedulers. > > In the days before the SMP enabled Erlang emulator (VM) we would need to > run multiple nodes (VM) to take advantage of multiple cores. Now the > default mode for the VM is to run as many schedulers as there are cores > available. This also highly configurable. > > Although Erlang's distribution allows processes to transparently > communicate across nodes there is a great performance advantage in > intra-node messaging as there is reference passing and it avoids > serialization in the external term format. > > It's interesting though how everything goes in cycles. Five years ago I > was keeping the core counts very high, and the node counts very low, to get > great performance. Now with cloud native we run a node in a container and > communicate with other (microservices) nodes using either distribution or > external protocols (e.g. REST) so the overheads are again much higher. But > the old way was vertical scaling and this is horizontal scaling, we pay a > tax but go from dozens of cores to hundreds or thousands. > > > On Wed, Jun 13, 2018, 08:46 joe mcguckin wrote: > >> >> If erlang is one large unix process with hundreds or thousands of it's >> own processes internally, how does Erlang make maximum use of a multi-core >> cpu? >> >> Wouldn?t Erlang be scheduled and run on a single core ? >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fxn@REDACTED Wed Jun 13 14:10:50 2018 From: fxn@REDACTED (Xavier Noria) Date: Wed, 13 Jun 2018 14:10:50 +0200 Subject: [erlang-questions] docs on group leaders? In-Reply-To: References: Message-ID: On Wed, Apr 4, 2018 at 2:32 PM, Xavier Noria wrote: what is the point of having each application master become the group leader > of its application? > For the archives, this is now documented: https://github.com/erlang/otp/pull/1771. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Wed Jun 13 14:54:02 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 13 Jun 2018 14:54:02 +0200 Subject: [erlang-questions] gen_statem state enter call and changing t oanother state In-Reply-To: <54e8714b-5c76-accb-ffa1-90a802434320@gmail.com> References: <20180611084533.GA61770@erix.ericsson.se> <20180611151543.GA85921@erix.ericsson.se> <20180611165539.GA8817@ferdmbp.local> <20180612080628.GB53065@erix.ericsson.se> <9d7e8d8d-f58e-c0af-f371-c42e584a83aa@gmail.com> <20180613010343.GC8817@ferdmbp.local> <54e8714b-5c76-accb-ffa1-90a802434320@gmail.com> Message-ID: <20180613125402.GA85302@erix.ericsson.se> On Wed, Jun 13, 2018 at 10:25:06AM +0200, Frans Schneider wrote: > Well, that was an enlightening discussion. Just ditched the initial > implementation and started all over again. Conditions now are true > conditions and events are real events. The number of states now is half > the number they were, but the handle_event functions tend to be more > complex. I could remove the whole state enter functionality. Oh... That's good I guess...? Enlightening discussion - agreed. I have started to look at how changing state_calls into special events instead of special calls would look in the code. It might end up cleaner to reason about. / Raimo > > Thanks, > > Frans > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From gattushivakrishna@REDACTED Wed Jun 13 15:03:58 2018 From: gattushivakrishna@REDACTED (shiva) Date: Wed, 13 Jun 2018 18:33:58 +0530 Subject: [erlang-questions] Query on odbc driver Message-ID: <6ecaf6fc-abcb-33a0-7cbf-4fd1e5a4fb67@utl.in> Hello Erlang team, I got a requirement to use Postgresql Database as part of our Erlang web application. To communicate from Erlang to Database I need a connector. Erlang provides odbc driver to connect but still few client libraries(epgsql,pgsql,etc.,) are available to communicate with Postgresql. What is the difference between odbc driver and other client libraries. Can anyone suggest which is the best one to use.? Thanks in advance. Thanks and regards, Gattu Shivakrishna. From raimo+erlang-questions@REDACTED Wed Jun 13 15:08:54 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 13 Jun 2018 15:08:54 +0200 Subject: [erlang-questions] missing line numbers when gen_statem crashes In-Reply-To: References: Message-ID: <20180613130854.GB85302@erix.ericsson.se> On Wed, Jun 13, 2018 at 11:51:31AM +0200, Torben Hoffmann wrote: > Hi, > > I have a crashing gen_statem, but to the best of my knowledge I cannot see > from the stock error messages which line my gen_statem dies on. > > I have searched high and low in my console output, but this is the only > thing that I could find: > > =CRASH REPORT==== 13-Jun-2018::11:17:28 === > crasher: > initial call: eee_sched_shard:init/1 > pid: <0.3149.0> > registered_name: [] > exception error: {badrecord,trans_opts} > in function gen_statem:loop_event_actions_list/10 (gen_statem.erl, > line 1210) This is an internal crash in gen_statem - I think i recognize this badrecord at line 1210. Your callback module returns a bad action in the actions list, and when gen_statem terminates there is a bug that causes it to crash instead, hiding the informative reason. It might be so that you return an action that is not allowed from a state enter call, or simply a not recognized action. Since the error is "cosmetic" or rather "not in the function", I did not patch this in the 20.3 track. The bad diagnostics error was introduced in 20.3 and is corrected in 21.0-rc1 and forwards. > ancestors: [<0.3148.0>,<0.3114.0>,<0.3107.0>,eee_sched_streams_sup, > eee_sched_task_sup,eee_sched_sup,<0.2867.0>] > message_queue_len: 0 > messages: [] > links: [<0.3148.0>] > dictionary: [] > trap_exit: false > status: running > heap_size: 6772 > stack_size: 27 > reductions: 58869 > neighbours: > > I'm about to turn on debug mode, but I don't think that that should be the > answer to this kind of problem. Line numbers should come out by default. In this case no line number can be presented since the callback module returned a bad term, and it is when processing the return value that the error is discovered. In this case the bad action term should be presented in the exception error (which it is from 21.0-rc1). > > Cheers, > Torben > -- > https://www.linkedin.com/in/torbenhoffmann/ -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From frank.muller.erl@REDACTED Wed Jun 13 15:09:30 2018 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 13 Jun 2018 15:09:30 +0200 Subject: [erlang-questions] [ANN] Asciideck: Asciidoc for Erlang In-Reply-To: <55342ba7-7313-db7f-d991-710659db3812@ninenines.eu> References: <55342ba7-7313-db7f-d991-710659db3812@ninenines.eu> Message-ID: Great work, and thanks for sharing!!! /Frank Hello, > > In my spare time I wrote an Asciidoc parser and translator project: > > https://github.com/ninenines/asciideck > > As far as I can tell this is the first Asciidoc implementation on BEAM. > > Three strong points: > > * It's not Markdown; Asciidoc is better defined and more powerful > * The parser gives you an AST that you can further manipulate > * The code is very readable, this is the code for parsing section titles > for example: > > https://github.com/ninenines/asciideck/blob/master/src/asciideck_block_parser.erl#L134 > - let it crash! > > I've switched ninenines.eu to use it and it builds the docs of all my > projects. It also builds man pages of the projects. More work remains to > be done but it's in a very good shape already. > > I've written about it in more details at: > > https://ninenines.eu/articles/asciideck/ > > Cheers, > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From torben.lehoff@REDACTED Wed Jun 13 16:03:17 2018 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Wed, 13 Jun 2018 16:03:17 +0200 Subject: [erlang-questions] missing line numbers when gen_statem crashes In-Reply-To: <20180613130854.GB85302@erix.ericsson.se> References: <20180613130854.GB85302@erix.ericsson.se> Message-ID: Hi Raimo, That was indeed the case. Was missing a From in a reply tuple. I wonder what can be done to make all of this more approachable. My memory of working with the gen_fsm is a much more pleasant one when you did errors. I can appreciate that stuffing things into the callback return tuples makes things harder, but it goes against one of the best things about OTP: it has always been super easy to find the place in the code where you have messed up. With gen_statem I'm not getting the same kind of tranquility as with gen_server and gen_statem. I'm wondering if the replies and next_event's should be made explicit calls in the code. That way you'd get a line number in your face if you did it wrong. And it would simplify the callback return structures quite a bit. So {reply, From, Msg} would have to be done with gen_statem:reply/1,2 exclusively and {next_event, Type, EventContent} would need new functions gen_statem:next_event/1,2. If the gen_statem:next_event/1,2 was a thing one could do this without removing the complex callback return values. Now that I know what pain error in the callback return values cause when debugging I would switch to using them exclusively, if the next_event/1,2 functions were available. I value ease of debugging that much. Cheers, Torben On Wed, Jun 13, 2018 at 3:09 PM Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > On Wed, Jun 13, 2018 at 11:51:31AM +0200, Torben Hoffmann wrote: > > Hi, > > > > I have a crashing gen_statem, but to the best of my knowledge I cannot > see > > from the stock error messages which line my gen_statem dies on. > > > > I have searched high and low in my console output, but this is the only > > thing that I could find: > > > > =CRASH REPORT==== 13-Jun-2018::11:17:28 === > > crasher: > > initial call: eee_sched_shard:init/1 > > pid: <0.3149.0> > > registered_name: [] > > exception error: {badrecord,trans_opts} > > in function gen_statem:loop_event_actions_list/10 (gen_statem.erl, > > line 1210) > > This is an internal crash in gen_statem - I think i recognize this > badrecord at line 1210. Your callback module returns a bad action in the > actions list, and when gen_statem terminates there is a bug that causes it > to crash instead, hiding the informative reason. > > It might be so that you return an action that is not allowed from a state > enter call, or simply a not recognized action. > > Since the error is "cosmetic" or rather "not in the function", I did not > patch this in the 20.3 track. The bad diagnostics error was introduced in > 20.3 and is corrected in 21.0-rc1 and forwards. > > > > ancestors: [<0.3148.0>,<0.3114.0>,<0.3107.0>,eee_sched_streams_sup, > > eee_sched_task_sup,eee_sched_sup,<0.2867.0>] > > message_queue_len: 0 > > messages: [] > > links: [<0.3148.0>] > > dictionary: [] > > trap_exit: false > > status: running > > heap_size: 6772 > > stack_size: 27 > > reductions: 58869 > > neighbours: > > > > I'm about to turn on debug mode, but I don't think that that should be > the > > answer to this kind of problem. Line numbers should come out by default. > > In this case no line number can be presented since the callback module > returned a bad term, and it is when processing the return value that the > error is discovered. In this case the bad action term should be presented > in the exception error (which it is from 21.0-rc1). > > > > > > Cheers, > > Torben > > -- > > https://www.linkedin.com/in/torbenhoffmann/ > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- https://www.linkedin.com/in/torbenhoffmann/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fchschneider@REDACTED Wed Jun 13 19:08:25 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Wed, 13 Jun 2018 19:08:25 +0200 Subject: [erlang-questions] list comprehension abuse Message-ID: Dear list, Just stumbled over some code using the following construct: [do_something(X) || X /= 1, X /= 3] Works rather nice, but is this considered good programming? I think it is called "list comprehension with no generator" and has to do something with OTP-7846. Frans From elbrujohalcon@REDACTED Wed Jun 13 19:11:58 2018 From: elbrujohalcon@REDACTED (Brujo Benavides) Date: Wed, 13 Jun 2018 14:11:58 -0300 Subject: [erlang-questions] list comprehension abuse In-Reply-To: References: Message-ID: I wrote a blog post about this a while back: https://medium.com/erlang-battleground/list-incomprehensions-ec504ec927c5 Cheers! :) Brujo Benavides > On 13 Jun 2018, at 14:08, Frans Schneider wrote: > > Dear list, > > Just stumbled over some code using the following construct: > > [do_something(X) || X /= 1, X /= 3] > > Works rather nice, but is this considered good programming? I think it is called "list comprehension with no generator" and has to do something with OTP-7846. > > Frans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonas.falkevik@REDACTED Wed Jun 13 23:08:07 2018 From: jonas.falkevik@REDACTED (Jonas Falkevik) Date: Wed, 13 Jun 2018 23:08:07 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> Message-ID: <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> Hi, I think I was able to reproduce this problem. And to me it looks like a bug. On my system the packet_inet_input is called but the recv call returns eagain. This isn?t freeing the buffer so the realloc (intended for fragments?) are trigged. Reallocating the same size buffer again. The below branch with the below change reduced the reallocs on my linux system. https://github.com/falkevik/otp/tree/udp_performance_bug https://github.com/falkevik/otp/commit/ca08458a0663792390d5e89f83b883e19a33054c Jonas > On 31 May 2018, at 21:05, Max Lapshin wrote: > > It is video, we need to receive packets strictly in order. > > Yes, I know that UDP can reorder and lose packets, but in normal conditions with hardware headend and plain C linux software on receiver it is possible to achieve almost 100% of deliverability of packets without reordering. > > > We want to achieve something like this with erlang. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jordanchaitin@REDACTED Thu Jun 14 05:28:31 2018 From: jordanchaitin@REDACTED (JC) Date: Thu, 14 Jun 2018 08:58:31 +0530 Subject: [erlang-questions] Query on odbc driver In-Reply-To: <6ecaf6fc-abcb-33a0-7cbf-4fd1e5a4fb67@utl.in> References: <6ecaf6fc-abcb-33a0-7cbf-4fd1e5a4fb67@utl.in> Message-ID: <66ea47fb-f95f-4a35-cfa1-0531e21c48b2@gmail.com> I use Erlang with Postgres regularly. In addition to the driver, one generally also needs a pooling tool on both the Postgres side, as well as the Erlang side. Pgapp is a commonly used front end with pool support, (although, IMO, a bit heavy) https://github.com/epgsql/pgapp Epgsql had many forks in the past, but this one seems to be the most well maintained and commonly used library now https://github.com/epgsql/epgsql Episcina https://github.com/erlware/episcina is another front end with pooling support, although it doesn't seem to work anymore, and the project seems unmaintained at this point. Semiocast's pgsql https://github.com/semiocast/pgsql is a good tool too. PGO https://github.com/SpaceTime-IoT/pgo is a new but neat little tool combining pooling and a driver in one, and works quite well for standard usage (there might be some unhandled edge cases). I believe pgo's author intends to replace the backend with the more popular Epgsql to avoid duplication of work on the protocol side. I have used Epgsql + pgapp, as well as Pgsql + Episcina in the past. At the present time I need to do mostly only parametrized queries and I use pgo. For simple needs, I would recommend PGO. For more complex use cases, go for Epgsql + Pgapp. Cheers On 6/13/18 18:33, shiva wrote: > Hello Erlang team, > > I got a requirement to use Postgresql Database as part of our Erlang > web application. To communicate from Erlang to Database I need a > connector. Erlang provides odbc driver to connect but still few client > libraries(epgsql,pgsql,etc.,) are available to communicate with > Postgresql. What is the difference between odbc driver and other > client libraries. Can anyone suggest which is the best one to use.? > > Thanks in advance. > > > Thanks and regards, > > Gattu Shivakrishna. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From john.hogberg@REDACTED Thu Jun 14 09:32:47 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Thu, 14 Jun 2018 07:32:47 +0000 Subject: [erlang-questions] Patch package OTP 20.3.8 released Message-ID: <1528961567.5620.2.camel@ericsson.com> Patch Package:???????????OTP 20.3.8 Git Tag:?????????????????OTP-20.3.8 Date:????????????????????2018-06-14 Trouble Report Id:???????OTP-14196, OTP-15109 Seq num:?????????????????ERIERL-161 System:??????????????????OTP Release:?????????????????20 Application:?????????????erts-9.3.3, snmp-5.2.11 Predecessor:?????????????OTP 20.3.7 ?Check out the git tag OTP-20.3.8, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- erts-9.3.3 ------------------------------------------------------ ?--------------------------------------------------------------------- ?The erts-9.3.3 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15109????Application(s): erts ???????????????Fixed bug in ets that could cause VM crash if process A ???????????????terminates after fixating a table and process B deletes ???????????????the table at "the same time". The table fixation could ???????????????be done with ets:safe_fixtable or if process A ???????????????terminates in the middle of a long running select or ???????????????match call. ?Full runtime dependencies of erts-9.3.3: kernel-5.0, sasl-3.0.1, ?stdlib-3.0 ?--------------------------------------------------------------------- ?--- snmp-5.2.11 ----------------------------------------------------- ?--------------------------------------------------------------------- ?The snmp-5.2.11 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-14196????Application(s): snmp ???????????????Related Id(s): ERIERL-161 ???????????????The Snmp MIB compiler now allows using a ???????????????TEXTUAL-CONVENTION type before defining it. ?Full runtime dependencies of snmp-5.2.11: crypto-3.3, erts-6.0, ?kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, stdlib-2.5 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- From rickard@REDACTED Thu Jun 14 17:06:41 2018 From: rickard@REDACTED (Rickard Green) Date: Thu, 14 Jun 2018 17:06:41 +0200 Subject: [erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine In-Reply-To: References: Message-ID: On Mon, May 7, 2018 at 12:06 PM, Kenneth Lundin wrote: > > > On Mon, May 7, 2018 at 8:54 AM, Jos? Valim wrote: > >> >> What about documentation from the OTP team that outlines a deprecation >> policy? It may be a good opportunity to also outline other compatibility >> guarantees, such as the compiled bytecode guarantees and the node >> compatibility guarantees, if such are not yet documented. >> > > I think it is a good idea to describe the OTP policy regarding > deprecation, backward compatibility, supported releases etc. in one place. > Will try to have it available before the OTP 21 release. > > Document available in PR-1839 Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From dane@REDACTED Thu Jun 14 19:38:13 2018 From: dane@REDACTED (Maxim Fedorov) Date: Thu, 14 Jun 2018 17:38:13 +0000 Subject: [erlang-questions] High Priority processes not handling system tasks In-Reply-To: References: Message-ID: Maybe it?s worth having some timeout in receive loop of cpc_receive? Otherwise erts_code_purger gets completely stuck, this code_server get_server stops responding too, and it leads to problems pretty much everywhere. That said, a single high-priority process in a tight loop stops all other processes, including those of max priority. Regards, Maxim Fedorov From: on behalf of Rickard Green Date: Saturday, May 26, 2018 at 10:59 To: "erlang-questions@REDACTED" , Maxim Fedorov Subject: Re: High Priority processes not handling system tasks I don't know why this mail only ended up in the archive, but not in my mailbox... > G?Day, > Below is a chunk of code that does not look bad at a first > glance, but? > > start() -> > Pid = spawn(?MODULE, high, []), > io:format("Starting... "), > erlang:check_process_code(Pid, rec), > io:format("collected~n"). > > high() -> > erlang:process_flag(priority, high), > endless(10). > > endless(X) -> > endless(X+1). > > > start() never returns, because check_process_code schedules a normal priority > system task against high priority process. It seems to be logical, but leads to > some really weird effects. For example, code:soft_purge() hangs (in > erts_code_purger, cpc_receive), thus whole code_server gen_server hangs, > and this paralyses almost everything (a separate issue - cpc_receive has no > receive timeout at all, possibly stalling whole VM). > > Is it designed behaviour? I have quite a straightforward solution (patch for > erl_process.c), but I might now understand consequences. > > Applies to R19+ (with 'new purge strategy' enabled). Older releases (eg. R16B) > are not affected. > > > Regards, > > Maxim Fedorov Yes this is how the process priority scheme was intentionally designed (for better or for worse). 'high' and 'max' process priority work are alway selected before lower process priority work. Changing the behaviour of system tasks so that lower priority tasks are interleaved with 'high' and/or 'max' priority work essentially introduce intentional priority inversion. Utilization of 'high' and 'max' priority has to be done with outmost care. Using high or max priorities on processes that hogs the cpu very likely will get you into trouble. The documentation of process_flag(priority, _) is also quite clear on this. This question comes up every now and then. In the past we have always rejected proposals for priory tweeks like this. I'm not claiming this priority scheme is the ultimate scheme, but this is what we have and I think we should stick with it as it is or redesign it from scratch. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Fri Jun 15 08:38:24 2018 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 15 Jun 2018 08:38:24 +0200 Subject: [erlang-questions] missing line numbers when gen_statem crashes In-Reply-To: References: <20180613130854.GB85302@erix.ericsson.se> Message-ID: <20180615063824.GA38923@erix.ericsson.se> On Wed, Jun 13, 2018 at 04:03:17PM +0200, Torben Hoffmann wrote: > Hi Raimo, > > That was indeed the case. Was missing a From in a reply tuple. > > I wonder what can be done to make all of this more approachable. > My memory of working with the gen_fsm is a much more pleasant one when you > did errors. > I can appreciate that stuffing things into the callback return tuples makes > things harder, but it goes against one of the best things about OTP: it has > always been super easy to find the place in the code where you have messed > up. > With gen_statem I'm not getting the same kind of tranquility as with > gen_server and gen_statem. This particular case was a bug hiding the intended error message. In OTP-21.0 you will get a regular error report (that contains among other things the current state, the event in play and the reason: {bad_action_from_state_function,Action} For a bad return tuple you would get: {bad_return_from_state_function,Result} For what is not allowed from a state enter call you get: {bad_state_enter_action_from_state_function,Action} and {bad_state_enter_return_from_state_function,Result} There are also these: {bad_return_from_init,Result} {bad_return_from_callback_mode,CallbackMode} {bad_reply_action_from_state_function,R} I hope they will be more helpful and enough to pinpoint the cause. The current state and event is not exactly the same as the callback module line number, but should be about as useful. > > I'm wondering if the replies and next_event's should be made explicit calls > in the code. > That way you'd get a line number in your face if you did it wrong. > And it would simplify the callback return structures quite a bit. > > So {reply, From, Msg} would have to be done with gen_statem:reply/1,2 > exclusively and {next_event, Type, EventContent} would need new functions > gen_statem:next_event/1,2. That does not play well with sys:trace/2 debugging, since the debug state is in the behaviour engine context and not reachable when called from the callback module. That kind of model is very non-functional but could be done by passing values through the process dictionary from the callback module context to the behaviour engine context. I do not like that at all. > > If the gen_statem:next_event/1,2 was a thing one could do this without > removing the complex callback return values. > Now that I know what pain error in the callback return values cause when You keep talking about the pain increased by a bug corrected in 21.0. Please re-evaluate the 21.0 version to see how painful it hopefully not is. / Raimo > debugging I would switch to using them exclusively, if the next_event/1,2 > functions were available. I value ease of debugging that much. > > Cheers, > Torben > > On Wed, Jun 13, 2018 at 3:09 PM Raimo Niskanen < > raimo+erlang-questions@REDACTED> wrote: > > > On Wed, Jun 13, 2018 at 11:51:31AM +0200, Torben Hoffmann wrote: > > > Hi, > > > > > > I have a crashing gen_statem, but to the best of my knowledge I cannot > > see > > > from the stock error messages which line my gen_statem dies on. > > > > > > I have searched high and low in my console output, but this is the only > > > thing that I could find: > > > > > > =CRASH REPORT==== 13-Jun-2018::11:17:28 === > > > crasher: > > > initial call: eee_sched_shard:init/1 > > > pid: <0.3149.0> > > > registered_name: [] > > > exception error: {badrecord,trans_opts} > > > in function gen_statem:loop_event_actions_list/10 (gen_statem.erl, > > > line 1210) > > > > This is an internal crash in gen_statem - I think i recognize this > > badrecord at line 1210. Your callback module returns a bad action in the > > actions list, and when gen_statem terminates there is a bug that causes it > > to crash instead, hiding the informative reason. > > > > It might be so that you return an action that is not allowed from a state > > enter call, or simply a not recognized action. > > > > Since the error is "cosmetic" or rather "not in the function", I did not > > patch this in the 20.3 track. The bad diagnostics error was introduced in > > 20.3 and is corrected in 21.0-rc1 and forwards. > > > > > > > ancestors: [<0.3148.0>,<0.3114.0>,<0.3107.0>,eee_sched_streams_sup, > > > eee_sched_task_sup,eee_sched_sup,<0.2867.0>] > > > message_queue_len: 0 > > > messages: [] > > > links: [<0.3148.0>] > > > dictionary: [] > > > trap_exit: false > > > status: running > > > heap_size: 6772 > > > stack_size: 27 > > > reductions: 58869 > > > neighbours: > > > > > > I'm about to turn on debug mode, but I don't think that that should be > > the > > > answer to this kind of problem. Line numbers should come out by default. > > > > In this case no line number can be presented since the callback module > > returned a bad term, and it is when processing the return value that the > > error is discovered. In this case the bad action term should be presented > > in the exception error (which it is from 21.0-rc1). > > > > > > > > > > Cheers, > > > Torben > > > -- > > > https://www.linkedin.com/in/torbenhoffmann/ > > > > > > -- > > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > https://www.linkedin.com/in/torbenhoffmann/ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From publicityifl@REDACTED Fri Jun 15 10:45:03 2018 From: publicityifl@REDACTED (Jurriaan Hage) Date: Fri, 15 Jun 2018 04:45:03 -0400 Subject: [erlang-questions] Call for draft papers for presentation at IFL 2018 (Implementation and Application of Functional Languages) Message-ID: Hello, Please, find below the fourth call for papers for IFL 2018. Please forward these to anyone you think may be interested. Apologies for any duplicates you may receive. best regards, Jurriaan Hage Publicity Chair of IFL --- Call for Draft papers for presentations ================================================================================ IFL 2018 30th Symposium on Implementation and Application of Functional Languages University of Massachusetts Lowell, MA, USA September 5th-7th, 2018 http://iflconference.org ================================================================================ ### Scope The goal of the IFL symposia is to bring together researchers actively engaged in the implementation and application of functional and function-based programming languages. IFL 2018 will be a venue for researchers to present and discuss new ideas and concepts, work in progress, and publication-ripe results related to the implementation and application of functional languages and function-based programming. Topics of interest to IFL include, but are not limited to: - language concepts - type systems, type checking, type inferencing - compilation techniques - staged compilation - run-time function specialization - run-time code generation - partial evaluation - (abstract) interpretation - metaprogramming - generic programming - automatic program generation - array processing - concurrent/parallel programming - concurrent/parallel program execution - embedded systems - web applications - (embedded) domain specific languages - security - novel memory management techniques - run-time profiling performance measurements - debugging and tracing - virtual/abstract machine architectures - validation, verification of functional programs - tools and programming techniques - (industrial) applications ### Keynote Speakers * Adam Chlipala, Massachusetts Institute of Technology CSAIL * Arjun Guha, University of Massachusetts Amherst ### Submissions and peer-review Differently from previous editions of IFL, IFL 2018 solicits two kinds of submissions: * Regular papers (12 pages including references) * Draft papers for presentations ('weak' limit between 8 and 15 pages) Regular papers will undergo a rigorous review by the program committee, and will be evaluated according to their correctness, novelty, originality, relevance, significance, and clarity. A set of regular papers will be conditionally accepted for publication. Authors of conditionally accepted papers will be provided with committee reviews along with a set of mandatory revisions. Regular papers not accepted for publication will be considered as draft papers, at the request of the author. Draft papers will be screened to make sure that they are within the scope of IFL, and will be accepted for presentation or rejected accordingly. Prior to the symposium: Authors of conditionally accepted papers and accepted presentations will submit a pre-proceedings version of their work that will appear in the draft proceedings distributed at the symposium. The draft proceedings does not constitute a formal publication. We require that at least one of the authors present the work at IFL 2018. After the symposium: Authors of conditionally accepted papers will submit a revised versions of their paper for the formal post-proceedings. The program committee will assess whether the mandatory revisions have been adequately addressed by the authors and thereby determines the final accept/reject status of the paper. Our interest is to ultimately accept all conditionally accepted papers. If you are an author of a conditionally accepted paper, please make sure that you address all the concerns of the reviewers. Authors of accepted presentations will be given the opportunity to incorporate the feedback from discussions at the symposium and will be invited to submit a revised full article for the formal post-proceedings. The program committee will evaluate these submissions according to their correctness, novelty, originality, relevance, significance, and clarity, and will thereby determine whether the paper is accepted or rejected. ### Publication The formal proceedings will appear in the International Conference Proceedings Series of the ACM Digital Library. At no time may work submitted to IFL be simultaneously submitted to other venues; submissions must adhere to ACM SIGPLAN's republication policy: http://www.sigplan.org/Resources/Policies/Republication ### Important dates Submission of regular papers: May 25, 2018 [PASSED!] Submission of draft papers: July 17, 2018 [UPCOMING!] Regular and draft papers notification: July 20, 2018 Deadline for early registration: August 8, 2018 Submission of pre-proceedings version: August 29, 2018 IFL Symposium: September 5-7, 2018 Submission of papers for post-proceedings: November 7, 2018 Notification of acceptance: December 22, 2018 Camera-ready version: February 10, 2019 ### Submission details All contributions must be written in English. Papers must use the ACM two columns conference format, which can be found at: http://www.acm.org/publications/proceedings-template Authors submit through EasyChair: https://easychair.org/conferences/?conf=ifl2018 ### Peter Landin Prize The Peter Landin Prize is awarded to the best paper presented at the symposium every year. The honored article is selected by the program committee based on the submissions received for the formal review process. The prize carries a cash award equivalent to 150 Euros. ### Organization and Program committee Chairs: Jay McCarthy & Matteo Cimini, University of Massachusetts Lowell, USA Program Committee: * Arthur Chargueraud, Inria, FR * Ben Delaware, Purdue University, USA * Christos Dimoulas, Northwestern University, USA * David Darais, University of Vermont, USA * Dominic Orchard, University of Kent, UK * Ekaterina Komendantskaya, Heriot-Watt University, UK * Garrett Morris, University of Kansas, USA * Heather Miller, EPFL & Northeastern University, CH & USA * Jeremy Yallop, University of Cambridge, UK * Keiko Nakata, SAP Innovation Center Potsdam, DE * Laura Castro, University of A Coruna, ESP * Magnus Myreen, Chalmers University of Technology, SWE * Natalia Chechina, Bournemouth University, UK * Peter Achten, Radboud Universiteit Nijmegen, NL * Peter-Michael Osera, Grinnell College, USA * Richard Eisenberg, Bryn Mawr College, USA * Trevor McDonell, University of New South Wales, AUS * Yukiyoshi Kameyama, University of Tsukuba, JAP ### Venue The 30th IFL is organized by the University of Massachusetts Lowell. The City of Lowell is located at the heart of the Merrimack Valley just 30 miles northwest of Boston. Lowell can be easily reached by train or taxi. See the website for more information on the venue. ### Acknowledgments This call-for-papers is an adaptation and evolution of content from previous instances of IFL. We are grateful to prior organizers for their work, which is reused here. A part of IFL 2018 format and CFP language that describes conditionally accepted papers has been adapted from call-for-papers of OOPSLA conferences. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Jun 15 11:49:35 2018 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 15 Jun 2018 11:49:35 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> Message-ID: Hello! On Wed, Jun 13, 2018 at 11:08 PM Jonas Falkevik < jonas.falkevik@REDACTED> wrote: > Hi, > I think I was able to reproduce this problem. > And to me it looks like a bug. > On my system the packet_inet_input is called but the recv call returns > eagain. > This isn?t freeing the buffer so the realloc (intended for fragments?) are > trigged. > Reallocating the same size buffer again. > > The below branch with the below change reduced the reallocs on my linux > system. > https://github.com/falkevik/otp/tree/udp_performance_bug > > > https://github.com/falkevik/otp/commit/ca08458a0663792390d5e89f83b883e19a33054c > > Interesting... I wonder if maybe it wouldn't be better to solve this problem in the code for realloc so that no copy is done when a realloc of the same size if issued... that way we solve it in all places instead of only in the inet_driver Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonas.falkevik@REDACTED Fri Jun 15 12:07:22 2018 From: jonas.falkevik@REDACTED (Jonas Falkevik) Date: Fri, 15 Jun 2018 12:07:22 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> Message-ID: <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> > > > Interesting... I wonder if maybe it wouldn't be better to solve this problem in the code for realloc so that no copy is done when a realloc of the same size if issued... that way we solve it in all places instead of only in the inet_driver > That sounds reasonable. :) Some extra pointer fiddling will still be done, setting the same values. But that is nothing in comparison. Why select indicates that the socket is ready for reading but would block seems to boil down to performance. Jonas From jonas.falkevik@REDACTED Fri Jun 15 22:53:57 2018 From: jonas.falkevik@REDACTED (Jonas Falkevik) Date: Fri, 15 Jun 2018 22:53:57 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On Fri, Jun 15, 2018 at 12:07 PM, Jonas Falkevik < jonas.falkevik@REDACTED> wrote: > > > > > > Interesting... I wonder if maybe it wouldn't be better to solve this > problem in the code for realloc so that no copy is done when a realloc of > the same size if issued... that way we solve it in all places instead of > only in the inet_driver > > > > Something like https://github.com/falkevik/otp/commit/444fb00ff2a9d1f40a8c66f48bea1cf3f07ca86c ? >From 444fb00ff2a9d1f40a8c66f48bea1cf3f07ca86c Mon Sep 17 00:00:00 2001 From: jonasf Date: Fri, 15 Jun 2018 15:55:38 +0200 Subject: [PATCH] erts realloc optimization same size optimize the case when realloc is called with same size --- erts/emulator/beam/erl_binary.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/erts/emulator/beam/erl_binary.h b/erts/emulator/beam/erl_binary.h index 7dfd0c273a9..9ba6aac1a0d 100644 --- a/erts/emulator/beam/erl_binary.h +++ b/erts/emulator/beam/erl_binary.h @@ -419,6 +419,8 @@ erts_bin_realloc_fnf(Binary *bp, Uint size) ErtsAlcType_t type = (bp->intern.flags & BIN_FLAG_DRV) ? ERTS_ALC_T_DRV_BINARY : ERTS_ALC_T_BINARY; ASSERT((bp->intern.flags & BIN_FLAG_MAGIC) == 0); + if (bp->orig_size == size) + return bp; if (bsize < size) /* overflow */ return NULL; nbp = erts_realloc_fnf(type, (void *) bp, bsize); @@ -436,6 +438,8 @@ erts_bin_realloc(Binary *bp, Uint size) ErtsAlcType_t type = (bp->intern.flags & BIN_FLAG_DRV) ? ERTS_ALC_T_DRV_BINARY : ERTS_ALC_T_BINARY; ASSERT((bp->intern.flags & BIN_FLAG_MAGIC) == 0); + if (bp->orig_size == size) + return bp; if (bsize < size) /* overflow */ erts_realloc_enomem(type, bp, size); nbp = erts_realloc_fnf(type, (void *) bp, bsize); -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Sun Jun 17 18:57:20 2018 From: g@REDACTED (Guilherme Andrade) Date: Sun, 17 Jun 2018 17:57:20 +0100 Subject: [erlang-questions] [ANN] deigma: Continuous event sampler Message-ID: Hello list, I would like to announce the release of deigma 1.0.0, an event sampler. It performs sampling of reported events within continuous one second windows. The sampling percentage is steadily adjusted so that the events that seep through are representative of what's happening in the system while not exceeding specified rate limits. The sampling percentage is also exposed in the context of each reported event, so that whichever other component that later receives the samples can perform reasonable guesses about the original population with limited information. * Overview: https://github.com/g-andrade/deigma/blob/master/README.md * Hex.pm package: https://hex.pm/packages/deigma * Documentation: https://hexdocs.pm/deigma/ * Source code: https://github.com/g-andrade/deigma -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Jun 18 15:57:14 2018 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 18 Jun 2018 15:57:14 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On Fri, Jun 15, 2018 at 10:54 PM Jonas Falkevik < jonas.falkevik@REDACTED> wrote: > > > On Fri, Jun 15, 2018 at 12:07 PM, Jonas Falkevik < > jonas.falkevik@REDACTED> wrote: > >> > >> > >> > Interesting... I wonder if maybe it wouldn't be better to solve this >> problem in the code for realloc so that no copy is done when a realloc of >> the same size if issued... that way we solve it in all places instead of >> only in the inet_driver >> > >> >> Something like > https://github.com/falkevik/otp/commit/444fb00ff2a9d1f40a8c66f48bea1cf3f07ca86c > ? > I was thinking something more like this: https://github.com/garazdawi/otp/tree/lukas/erts/realloc_thr_pref_shrink_th This makes it so that the absolute and relative single block shrink thesholds are respected for reallocs made on a remote scheduler. This should solve the problem that you have found also, through as I still haven't reproduced it I can't test that it actually solves it. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From me@REDACTED Mon Jun 18 18:32:03 2018 From: me@REDACTED (Nato) Date: Mon, 18 Jun 2018 09:32:03 -0700 Subject: [erlang-questions] forgETS project Message-ID: Does anyone have any privy information on the availability/status of the forgETS db project coming out of WhatsApp/FB? Is it OSS? -- Nato From bob@REDACTED Mon Jun 18 22:22:39 2018 From: bob@REDACTED (Bob Cowdery) Date: Mon, 18 Jun 2018 21:22:39 +0100 Subject: [erlang-questions] Help on priv_dir please Message-ID: <98afab68-9cde-88ab-09a7-c1d308887600@bobcowdery.plus.com> Hi, I'm trying to get gen_serial up and running. I've tracked the problem down to a line which boils down to code:priv_dir(gen_serial). This is intending to get the port executable path but returns bad_name. Why would this call fail? Thanks BobC From hawk.mattsson@REDACTED Mon Jun 18 22:59:52 2018 From: hawk.mattsson@REDACTED (=?UTF-8?Q?H=C3=A5kan_Mattsson?=) Date: Mon, 18 Jun 2018 22:59:52 +0200 Subject: [erlang-questions] Help on priv_dir please In-Reply-To: <98afab68-9cde-88ab-09a7-c1d308887600@bobcowdery.plus.com> References: <98afab68-9cde-88ab-09a7-c1d308887600@bobcowdery.plus.com> Message-ID: It looks like the gen_serial app not is in the code path. See what code:get_path() returns. /H?kan On Mon, Jun 18, 2018 at 10:22 PM, Bob Cowdery wrote: > Hi, > > I'm trying to get gen_serial up and running. I've tracked the problem down > to a line which boils down to code:priv_dir(gen_serial). > > This is intending to get the port executable path but returns bad_name. > Why would this call fail? > > Thanks > > BobC > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hinde@REDACTED Tue Jun 19 00:06:29 2018 From: sean.hinde@REDACTED (Sean Hinde) Date: Tue, 19 Jun 2018 00:06:29 +0200 Subject: [erlang-questions] forgETS project In-Reply-To: <20180618163211.DACD65C02A@hades.cslab.ericsson.net> References: <20180618163211.DACD65C02A@hades.cslab.ericsson.net> Message-ID: <24CDEF3D-4D09-4A0E-8641-6B2D12CFB04C@mac.com> Nothing private, but in the talk there was a clear statement that the plan was to make it open source. Also that there was a process to follow in Facebook which might take a while. I?m also looking forward to this one Sean > On 18 Jun 2018, at 18:32, Nato wrote: > > Does anyone have any privy information on the > availability/status of the forgETS db project > coming out of WhatsApp/FB? Is it OSS? > > -- > Nato > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From krzysztof.jurewicz@REDACTED Tue Jun 19 16:30:36 2018 From: krzysztof.jurewicz@REDACTED (Krzysztof Jurewicz) Date: Tue, 19 Jun 2018 16:30:36 +0200 Subject: [erlang-questions] =?utf-8?q?In-process_memory_sharing_doesn?= =?utf-8?q?=E2=80=99t_seem_to_work?= Message-ID: <878t7a4zn7.fsf@gmail.com> I have a potentially big, in-memory data structure which I need to maintain in two copies, master-slave. The slave however will be synchronized only periodically (currently once per second) and it may accept ephemeral changes (lost with each synchronization). The changes will be relatively minor (most of the data structure will be preserved intact). For real world context, see the quoted documentation at https://github.com/tendermint/abci/issues/274 . A natural way to handle this problem without bothering about efficiency would be to put the slave state in a separate process and to synchronize it periodically by sending a message from master containing current data dump. However sending messages (and spawning new processes) involves copying data, while making an assignment in the same process should allow to benefit from memory sharing. There seems to be not much documentation about sharing, but it is described at http://erlang.org/doc/efficiency_guide/processes.html#id70776 (?Loss of sharing?) and there is a discussion at http://erlang.org/pipermail/erlang-questions/2009-September/046436.html . The problem is that memory measurements don?t seem to confirm that sharing actually occurs. For demonstration I?ll use the hog module from http://erlang.org/pipermail/erlang-bugs/2007-November/000488.html . Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-10.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] Eshell V10.0 (abort with ^G) 1> c(hog). hog.erl:2: Warning: export_all flag enabled - all functions will be exported hog.erl:29: Warning: variable 'Tree' is unused {ok,hog} 2> MemoryFun = fun() -> round(memory(processes_used) / 1024 / 1024) end. #Fun 3> MemoryFun(). 4 4> T = hog:tree(25), ok. ok 5> erts_debug:size(T). 75 6> erts_debug:flat_size(T). 100663293 So far, so good. 7> MemoryFun(). 1814 This already looks suspicious. An interesting fact about erlang:memory/1 is that it looks completely unstable, as running it after a while gives a different result: 8> MemoryFun(). 5739 top meanwhile reports 20,3% memory usage of 7843892 KiB total, so it?s more close to the first result. Interesting things happen also when we force garbage collection: 9> garbage_collect(), MemoryFun(). 1814 Looks better, however after a while we get back the bigger result, but two calls more are needed: 10> MemoryFun(). 1814 11> MemoryFun(). 5739 Let?s now create a tuple which references the same tree multiple times: 12> X = {T, T, T, T}, ok. ok eheap_alloc: Cannot allocate 5668310376 bytes of memory (of type "heap"). Crash dump is being written to: erl_crash.dump...done The generated erl_crash.dump takes 6 gigabytes. It?s interesting that we get ?ok?, which suggests that there is sufficient memory for X and Erlang shell tries to allocate the huge number of bytes for something else. Also output of hog:main/0 is completely different than one in the original report: 1> hog:main(). Creating tree:: allocated 0 bytes, heap size 376 bytes Receiving tree:: allocated 0 bytes, heap size 233 bytes true Does it mean that memory sharing no longer works in Erlang? Or maybe it is a shell-related problem? From michal@REDACTED Tue Jun 19 16:57:19 2018 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Tue, 19 Jun 2018 16:57:19 +0200 Subject: [erlang-questions] In-process memory sharing =?utf-8?Q?doesn=E2=80=99t_?=seem to work In-Reply-To: <878t7a4zn7.fsf@gmail.com> References: <878t7a4zn7.fsf@gmail.com> Message-ID: Hi Krzysztof, You're running your tests in the shell which has sometimes slightly different characteristics from the compiled code. The shell runs an interpreter that has various performance overheards and additional data copying might be one of them. I'd suggest running the same tests from a regular, compiled module in a regular process (not the shell process) and comparing the results. Micha?. On 19 Jun 2018, 16:31 +0200, Krzysztof Jurewicz , wrote: > I have a potentially big, in-memory data structure which I need to maintain in two copies, master-slave. The slave however will be synchronized only periodically (currently once per second) and it may accept ephemeral changes (lost with each synchronization). The changes will be relatively minor (most of the data structure will be preserved intact). For real world context, see the quoted documentation at https://github.com/tendermint/abci/issues/274 . A natural way to handle this problem without bothering about efficiency would be to put the slave state in a separate process and to synchronize it periodically by sending a message from master containing current data dump. However sending messages (and spawning new processes) involves copying data, while making an assignment in the same process should allow to benefit from memory sharing. > > There seems to be not much documentation about sharing, but it is described at http://erlang.org/doc/efficiency_guide/processes.html#id70776 (?Loss of sharing?) and there is a discussion at http://erlang.org/pipermail/erlang-questions/2009-September/046436.html . The problem is that memory measurements don?t seem to confirm that sharing actually occurs. For demonstration I?ll use the hog module from http://erlang.org/pipermail/erlang-bugs/2007-November/000488.html . > > Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-10.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] > > Eshell V10.0 (abort with ^G) > 1> c(hog). > hog.erl:2: Warning: export_all flag enabled - all functions will be exported > hog.erl:29: Warning: variable 'Tree' is unused > {ok,hog} > 2> MemoryFun = fun() -> round(memory(processes_used) / 1024 / 1024) end. > #Fun > 3> MemoryFun(). > 4 > 4> T = hog:tree(25), ok. > ok > 5> erts_debug:size(T). > 75 > 6> erts_debug:flat_size(T). > 100663293 > > So far, so good. > > 7> MemoryFun(). > 1814 > > This already looks suspicious. An interesting fact about erlang:memory/1 is that it looks completely unstable, as running it after a while gives a different result: > > 8> MemoryFun(). > 5739 > > top meanwhile reports 20,3% memory usage of 7843892 KiB total, so it?s more close to the first result. > > Interesting things happen also when we force garbage collection: > > 9> garbage_collect(), MemoryFun(). > 1814 > > Looks better, however after a while we get back the bigger result, but two calls more are needed: > > 10> MemoryFun(). > 1814 > 11> MemoryFun(). > 5739 > > Let?s now create a tuple which references the same tree multiple times: > > 12> X = {T, T, T, T}, ok. > ok > eheap_alloc: Cannot allocate 5668310376 bytes of memory (of type "heap"). > > Crash dump is being written to: erl_crash.dump...done > > The generated erl_crash.dump takes 6 gigabytes. It?s interesting that we get ?ok?, which suggests that there is sufficient memory for X and Erlang shell tries to allocate the huge number of bytes for something else. > > Also output of hog:main/0 is completely different than one in the original report: > > 1> hog:main(). > Creating tree:: allocated 0 bytes, heap size 376 bytes > Receiving tree:: allocated 0 bytes, heap size 233 bytes > true > > Does it mean that memory sharing no longer works in Erlang? Or maybe it is a shell-related problem? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik.x.nord@REDACTED Tue Jun 19 17:07:31 2018 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Tue, 19 Jun 2018 15:07:31 +0000 Subject: [erlang-questions] OTP 21.0 has been released Message-ID: <1529420850.5596.2.camel@ericsson.com> Erlang/OTP 21 is a new major release with new features, improvements as well as incompatibilities. Potential Incompatibilities ????All Corba applications are now moved from the OTP repository ????A new Corba repository will be created https://github.com/erlang ????New applications ftp and tftp, moved from inets ????ssl no longer supports 3_DES cipher suites or RSA-key exchange cipher suites by default ????erlang:monitor on a primitive node (erl_interface, jinterface, etc) will no longer fail with badarg exception. Instead a monitor will be created, but it will only supervise the connection to the node. ?Highlights ?Erts: ????Enhanced IO scalability ????Support for usage of distribution controller processes for alternative transports, routing etc ????compact instructions on 64bit systems for code below 4GB 20% less memory for loaded code ????Rewrite of the efile-driver with NIFs and "Dirty schedulers" resulting in faster file operations ????non-smp VM removed ????link and monitor optimized for scalability ????os:getenv/putenv now work on thread-safe emulation. No longer in sync with libc getenv(3). Manual synchronization will be needed. Compiler: ????Misc compiler optimizations including contributions from the Elixir team resulting in 10% improvements in benchmarks ????"Tuple calls" have been removed from the run-time system. ????Code such as f({ok, Val}) -> {ok, Val} is now automatically rewritten to f({ok, Val} = Tuple) -> Tuple. this reduces code size, execution time, and removed GC pressure. ????More information in stacktrace from a number of operators ????erlang:get_stacktrace/0 deprecated to be replaced with try ... catch C:R:Stacktrace -> ... ????Creation of small maps with literal keys optimized. ????A new predifined macro `OTP_RELEASE` and preprocessor directives `- if` and `-elif` Security: ????DTLS is now supported in the SSL application ????Enhanced support for distribution over TLS ????"unsecure" ciphers removed from defaults in SSL and SSH. ????A new option value defined to facilitate implementing exec servers. Old option kept for compatibility, but now gives errors on stderror. Standard libraries: ????New API for logging, logger ????New uri_string module for parsing URIs according to "The standard" ????New function lists:search(list,fun/1) -> {ok, Value} | false ????Changed default behaviour of .erlang loading. escript, erlc, dialyzer and typer no longer load an .erlang at all. For more details see http://erlang.org/download/otp_src_21.0.readme For installation instructions please consult the README file that is part of the distribution. Pre build versions for Windows can be fetched here: http://erlang.org/download/otp_win32_21.0.exe http://erlang.org/download/otp_win64_21.0.exe Online documentation can be browsed here: http://erlang.org/documentation/doc-10.0/doc The Erlang/OTP source can also be found at GitHub in the official Erlang repository, Here: OTP-21.0 Please report any new issues via Erlang/OTPs puplic issue tracker https://bugs.erlang.org We want to thank all of you who sent us patches, suggestions and bug reports! Thank you for all your contributions! The Erlang/OTP team at Ericsson From bob@REDACTED Tue Jun 19 17:32:49 2018 From: bob@REDACTED (Bob Cowdery) Date: Tue, 19 Jun 2018 16:32:49 +0100 Subject: [erlang-questions] Binaries Message-ID: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> If I have a number, say 1450000001 and I want to represent that as a binary in the form ?<<16#14,16#50,16#00,16#00,16#01>> what's the best way. I'm not sure what list_to_binary(integer_to_list(1450000001)) which prints as <<"1450000001">> produces but I guess its 10 bytes not 5. BobC From pierrefenoll@REDACTED Tue Jun 19 17:38:41 2018 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Tue, 19 Jun 2018 17:38:41 +0200 Subject: [erlang-questions] Binaries In-Reply-To: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> Message-ID: The integer-unit bit syntax should help you. http://erlang.org/doc/programming_examples/bit_syntax.html#segments Cheers, -- Pierre Fenoll On Tue, 19 Jun 2018 at 17:33, Bob Cowdery wrote: > If I have a number, say 1450000001 and I want to represent that as a > binary in the form > > <<16#14,16#50,16#00,16#00,16#01>> what's the best way. > > I'm not sure what list_to_binary(integer_to_list(1450000001)) which > prints as <<"1450000001">> produces but I guess its 10 bytes not 5. > > BobC > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo@REDACTED Tue Jun 19 17:48:00 2018 From: hugo@REDACTED (Hugo Mills) Date: Tue, 19 Jun 2018 15:48:00 +0000 Subject: [erlang-questions] Binaries In-Reply-To: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> Message-ID: <20180619154800.GC28972@carfax.org.uk> On Tue, Jun 19, 2018 at 04:32:49PM +0100, Bob Cowdery wrote: > If I have a number, say 1450000001 and I want to represent that as a > binary in the form > > ?<<16#14,16#50,16#00,16#00,16#01>> what's the best way. > > I'm not sure what list_to_binary(integer_to_list(1450000001)) which > prints as <<"1450000001">> produces but I guess its 10 bytes not 5. This is basically just binary coded decimal, isn't it? You should be able to do this with a binary comprehension: 1> << <<(C-$0):4>> || C <- "12345678">>. <<18,52,86,120>> Hugo. -- Hugo Mills | Great oxymorons of the world, no. 7: hugo@REDACTED carfax.org.uk | The Simple Truth http://carfax.org.uk/ | PGP: E2AB1DE4 | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From sverker.eriksson@REDACTED Tue Jun 19 17:51:47 2018 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Tue, 19 Jun 2018 15:51:47 +0000 Subject: [erlang-questions] Binaries In-Reply-To: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> Message-ID: <1529423506.3487.7.camel@ericsson.com> If you mean 16#1450000001, then <<16#1450000001:40/big>> will give you a 5 byte (40 bits) big endian format <<16#14,16#50,16#00,16#00,16#01>> If you mean 1450000001 as a decimal integer, then my guess is you do not actually want that. integer_to_list produces ASCII characters. /Sverker On tis, 2018-06-19 at 16:32 +0100, Bob Cowdery wrote: > If I have a number, say 1450000001 and I want to represent that as a? > binary in the form > > ??<<16#14,16#50,16#00,16#00,16#01>> what's the best way. > > I'm not sure what list_to_binary(integer_to_list(1450000001)) which? > prints as <<"1450000001">> produces but I guess its 10 bytes not 5. > > BobC > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bob@REDACTED Tue Jun 19 18:22:42 2018 From: bob@REDACTED (Bob Cowdery) Date: Tue, 19 Jun 2018 17:22:42 +0100 Subject: [erlang-questions] Binaries In-Reply-To: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> Message-ID: <39377dbb-b142-e1d0-39ac-3b89913672b0@bobcowdery.plus.com> Thanks for all the suggestions. Still a little confused. The number is an integer, actually a frequency in Hz plus a command byte at the end which is being sent over a serial connection in hex format using gen_serial. This command works: gen_serial:bsend(P, <<16#14,16#50,16#00,16#00,16#01>>). where P is the open Port. However, when I use any of the methods to create a hex version they all end up with <<20,80,0,0,1>> which to my mind is the decimal equiv of above. If I fire that I get: 5> gen_serial:bsend(P,<<20,80,0,0,1>>). ** exception error: bad argument ???? in function? port_command/2 ??????? called as port_command(#Port<0.470>,[<<"d">>|<<20,80,0,0,1>>]) ???? in call from gen_serial:send/2 (gen_serial.erl, line 624) ???? in call from gen_serial:bsend/3 (gen_serial.erl, line 706) What is <<"d">> doing in there? On 6/19/2018 4:32 PM, Bob Cowdery wrote: > If I have a number, say 1450000001 and I want to represent that as a > binary in the form > > ?<<16#14,16#50,16#00,16#00,16#01>> what's the best way. > > I'm not sure what list_to_binary(integer_to_list(1450000001)) which > prints as <<"1450000001">> produces but I guess its 10 bytes not 5. > > BobC > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bob@REDACTED Tue Jun 19 18:35:57 2018 From: bob@REDACTED (Bob Cowdery) Date: Tue, 19 Jun 2018 17:35:57 +0100 Subject: [erlang-questions] Binaries In-Reply-To: <39377dbb-b142-e1d0-39ac-3b89913672b0@bobcowdery.plus.com> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> <39377dbb-b142-e1d0-39ac-3b89913672b0@bobcowdery.plus.com> Message-ID: I think something got messed up. It's not failing now but just does nothing. So the encoding on the wire is different for the two versions that are numerically the same. On 6/19/2018 5:22 PM, Bob Cowdery wrote: > Thanks for all the suggestions. Still a little confused. The number is > an integer, actually a frequency in Hz plus a command byte at the end > which is being sent over a serial connection in hex format using > gen_serial. > > This command works: gen_serial:bsend(P, > <<16#14,16#50,16#00,16#00,16#01>>). where P is the open Port. > > However, when I use any of the methods to create a hex version they > all end up with <<20,80,0,0,1>> which to my mind is the decimal equiv > of above. > > If I fire that I get: > > 5> gen_serial:bsend(P,<<20,80,0,0,1>>). > ** exception error: bad argument > ???? in function? port_command/2 > ??????? called as port_command(#Port<0.470>,[<<"d">>|<<20,80,0,0,1>>]) > ???? in call from gen_serial:send/2 (gen_serial.erl, line 624) > ???? in call from gen_serial:bsend/3 (gen_serial.erl, line 706) > > What is <<"d">> doing in there? > > > On 6/19/2018 4:32 PM, Bob Cowdery wrote: >> If I have a number, say 1450000001 and I want to represent that as a >> binary in the form >> >> ?<<16#14,16#50,16#00,16#00,16#01>> what's the best way. >> >> I'm not sure what list_to_binary(integer_to_list(1450000001)) which >> prints as <<"1450000001">> produces but I guess its 10 bytes not 5. >> >> BobC >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From z@REDACTED Tue Jun 19 18:37:15 2018 From: z@REDACTED (Danil Zagoskin) Date: Tue, 19 Jun 2018 19:37:15 +0300 Subject: [erlang-questions] Binaries In-Reply-To: <39377dbb-b142-e1d0-39ac-3b89913672b0@bobcowdery.plus.com> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> <39377dbb-b142-e1d0-39ac-3b89913672b0@bobcowdery.plus.com> Message-ID: That's how erlang part communicates with C part. https://github.com/tomszilagyi/gen_serial/blob/master/src/gen_serial.erl #L622 -define(PACKET_DATA, $d). send(#gen_serial{port = Port}, Data) -> true = port_command(Port, [<> | Data]), ok. Maybe port_command does not expect an improper list ? try one of these: * gen_serial:bsend(P,[<<20,80,0,0,1>>]) % Your binary wrapped in a list * gen_serial:bsend(P,[20,80,0,0,1]) % data as list instead of binary On Tue, Jun 19, 2018 at 7:22 PM Bob Cowdery wrote: > Thanks for all the suggestions. Still a little confused. The number is > an integer, actually a frequency in Hz plus a command byte at the end > which is being sent over a serial connection in hex format using > gen_serial. > > This command works: gen_serial:bsend(P, > <<16#14,16#50,16#00,16#00,16#01>>). where P is the open Port. > > However, when I use any of the methods to create a hex version they all > end up with <<20,80,0,0,1>> which to my mind is the decimal equiv of above. > > If I fire that I get: > > 5> gen_serial:bsend(P,<<20,80,0,0,1>>). > ** exception error: bad argument > in function port_command/2 > called as port_command(#Port<0.470>,[<<"d">>|<<20,80,0,0,1>>]) > in call from gen_serial:send/2 (gen_serial.erl, line 624) > in call from gen_serial:bsend/3 (gen_serial.erl, line 706) > > What is <<"d">> doing in there? > > > On 6/19/2018 4:32 PM, Bob Cowdery wrote: > > If I have a number, say 1450000001 and I want to represent that as a > > binary in the form > > > > <<16#14,16#50,16#00,16#00,16#01>> what's the best way. > > > > I'm not sure what list_to_binary(integer_to_list(1450000001)) which > > prints as <<"1450000001">> produces but I guess its 10 bytes not 5. > > > > BobC > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Tue Jun 19 18:53:44 2018 From: bob@REDACTED (Bob Cowdery) Date: Tue, 19 Jun 2018 17:53:44 +0100 Subject: [erlang-questions] Binaries In-Reply-To: References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> <39377dbb-b142-e1d0-39ac-3b89913672b0@bobcowdery.plus.com> Message-ID: <128ef2f6-eafe-d4b2-eeca-5c8b4afb477f@bobcowdery.plus.com> The first does nothing and the second hangs. It's not gen-serial that's having a problem with it. It's sending it and giving ok but when decoded in the radio which obviously I can't look at it isn't a valid command. The encoding of <<16#14,16#50,16#00,16#00,16#01>> and <<20,80,0,0,1>> is somehow different. On 6/19/2018 5:37 PM, Danil Zagoskin wrote: > That's how erlang part communicates with C part. > https://github.com/tomszilagyi/gen_serial/blob/master/src/gen_serial.erl#L622 > -define(PACKET_DATA, $d). > > send(#gen_serial{port = Port}, Data) -> > ? true = port_command(Port, [<> | Data]), > ? ok. > > Maybe port_command does not expect an improper list ? try one of these: > * gen_serial:bsend(P,[<<20,80,0,0,1>>]) % Your binary wrapped in a list > * gen_serial:bsend(P,[20,80,0,0,1]) % data as list instead of binary > > On Tue, Jun 19, 2018 at 7:22 PM Bob Cowdery > wrote: > > Thanks for all the suggestions. Still a little confused. The > number is > an integer, actually a frequency in Hz plus a command byte at the end > which is being sent over a serial connection in hex format using > gen_serial. > > This command works: gen_serial:bsend(P, > <<16#14,16#50,16#00,16#00,16#01>>). where P is the open Port. > > However, when I use any of the methods to create a hex version > they all > end up with <<20,80,0,0,1>> which to my mind is the decimal equiv > of above. > > If I fire that I get: > > 5> gen_serial:bsend(P,<<20,80,0,0,1>>). > ** exception error: bad argument > ????? in function? port_command/2 > ???????? called as > port_command(#Port<0.470>,[<<"d">>|<<20,80,0,0,1>>]) > ????? in call from gen_serial:send/2 (gen_serial.erl, line 624) > ????? in call from gen_serial:bsend/3 (gen_serial.erl, line 706) > > What is <<"d">> doing in there? > > > On 6/19/2018 4:32 PM, Bob Cowdery wrote: > > If I have a number, say 1450000001 and I want to represent that > as a > > binary in the form > > > > ?<<16#14,16#50,16#00,16#00,16#01>> what's the best way. > > > > I'm not sure what list_to_binary(integer_to_list(1450000001)) which > > prints as <<"1450000001">> produces but I guess its 10 bytes not 5. > > > > BobC > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Tue Jun 19 19:08:02 2018 From: bob@REDACTED (Bob Ippolito) Date: Tue, 19 Jun 2018 10:08:02 -0700 Subject: [erlang-questions] Binaries In-Reply-To: <128ef2f6-eafe-d4b2-eeca-5c8b4afb477f@bobcowdery.plus.com> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> <39377dbb-b142-e1d0-39ac-3b89913672b0@bobcowdery.plus.com> <128ef2f6-eafe-d4b2-eeca-5c8b4afb477f@bobcowdery.plus.com> Message-ID: <<16#14,16#50,16#00,16#00,16#01>> and <<20,80,0,0,1>> are identical. If you're experiencing any difference in behavior when using those expressions, your problem is really somewhere else. Perhaps the device is stateful and not getting properly reset from one call to the next, or the port P was closed because the process died with an exception. On Tue, Jun 19, 2018 at 9:53 AM Bob Cowdery wrote: > The first does nothing and the second hangs. It's not gen-serial that's > having a problem with it. It's sending it and giving ok but when decoded in > the radio which obviously I can't look at it isn't a valid command. The > encoding of <<16#14,16#50,16#00,16#00,16#01>> and <<20,80,0,0,1>> is > somehow different. > > On 6/19/2018 5:37 PM, Danil Zagoskin wrote: > > That's how erlang part communicates with C part. > https://github.com/tomszilagyi/gen_serial/blob/master/src/gen_serial.erl > #L622 > -define(PACKET_DATA, $d). > > send(#gen_serial{port = Port}, Data) -> > true = port_command(Port, [<> | Data]), > ok. > > Maybe port_command does not expect an improper list ? try one of these: > * gen_serial:bsend(P,[<<20,80,0,0,1>>]) % Your binary wrapped in a list > * gen_serial:bsend(P,[20,80,0,0,1]) % data as list instead of binary > > On Tue, Jun 19, 2018 at 7:22 PM Bob Cowdery > wrote: > >> Thanks for all the suggestions. Still a little confused. The number is >> an integer, actually a frequency in Hz plus a command byte at the end >> which is being sent over a serial connection in hex format using >> gen_serial. >> >> This command works: gen_serial:bsend(P, >> <<16#14,16#50,16#00,16#00,16#01>>). where P is the open Port. >> >> However, when I use any of the methods to create a hex version they all >> end up with <<20,80,0,0,1>> which to my mind is the decimal equiv of >> above. >> >> If I fire that I get: >> >> 5> gen_serial:bsend(P,<<20,80,0,0,1>>). >> ** exception error: bad argument >> in function port_command/2 >> called as port_command(#Port<0.470>,[<<"d">>|<<20,80,0,0,1>>]) >> in call from gen_serial:send/2 (gen_serial.erl, line 624) >> in call from gen_serial:bsend/3 (gen_serial.erl, line 706) >> >> What is <<"d">> doing in there? >> >> >> On 6/19/2018 4:32 PM, Bob Cowdery wrote: >> > If I have a number, say 1450000001 and I want to represent that as a >> > binary in the form >> > >> > <<16#14,16#50,16#00,16#00,16#01>> what's the best way. >> > >> > I'm not sure what list_to_binary(integer_to_list(1450000001)) which >> > prints as <<"1450000001">> produces but I guess its 10 bytes not 5. >> > >> > BobC >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > -- > Danil Zagoskin | z@REDACTED > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From krzysztof.jurewicz@REDACTED Tue Jun 19 21:16:51 2018 From: krzysztof.jurewicz@REDACTED (Krzysztof Jurewicz) Date: Tue, 19 Jun 2018 21:16:51 +0200 Subject: [erlang-questions] =?utf-8?q?In-process_memory_sharing_doesn?= =?utf-8?q?=E2=80=99t_seem_to_work?= In-Reply-To: References: <878t7a4zn7.fsf@gmail.com> Message-ID: <877emu4me4.fsf@gmail.com> > You're running your tests in the shell which has sometimes slightly different characteristics from the compiled code. The shell runs an interpreter that has various performance overheards and additional data copying might be one of them. I'd suggest running the same tests from a regular, compiled module in a regular process (not the shell process) and comparing the results. Thank you, it seems that this is indeed the case. Have added the following function to the hog module: create_and_hold_tree(N, Watcher) -> T = tree(N), Watcher ! ok, receive {release, To} -> To ! T end. 1> memory(processes_used) / 1024 / 1024. 4.1407012939453125 2> Holder = spawn(hog, create_and_hold_tree, [25, self()]). <0.79.0> 3> flush(). Shell got ok ok 4> memory(processes_used) / 1024 / 1024. 4.1668548583984375 5> Holder ! {release, self()}. {release,<0.76.0>} 6> memory(processes_used) / 1024 / 1024. 2692.868850708008 From zxq9@REDACTED Wed Jun 20 03:21:01 2018 From: zxq9@REDACTED (zxq9) Date: Wed, 20 Jun 2018 10:21:01 +0900 Subject: [erlang-questions] Binaries In-Reply-To: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> Message-ID: <2815626.v6yVoIg6k3@burrito> On 2018?6?19? ??? 16:32:49 Bob Cowdery wrote: > If I have a number, say 1450000001 and I want to represent that as a > binary in the form > > <<16#14,16#50,16#00,16#00,16#01>> what's the best way. > > I'm not sure what list_to_binary(integer_to_list(1450000001)) which > prints as <<"1450000001">> produces but I guess its 10 bytes not 5. Binary syntax can be a bit of a wonderland at first. The form you present above works just fine. Keep in mind that it yields the following: 1> <<16#14,16#50,16#00,16#00,16#01>>. <<20,80,0,0,1>> If we want to encode it using your original decimal representation (or as an integer programatically generated as a variable), and pack it into the same 40-bit space (5 8-bit bytes): 2> <<1450000001:40>>. <<0,86,109,62,129>> What, what? These aren't the same! There are two reasons: - The hexadecimal modifier 16# means 10#14 =/= 16#14 - Your version splits this integer value up into decimal-determined fields It appears that you don't actually want to encode the number 1450000001, but rather want five 8-bit fields carrying the values 14, 50, 00, 00, and 01, in that order. Is this correct? If you're starting with an integer that is *actually* 1,450,000,001 and you want to break it up by its decimal value then you'll want to div/rem it down by 100s to break it into chunks and then encode each chunk: 1> Split = fun 1> Chunk(0, Parts) -> Parts; 1> Chunk(Z, Parts) -> Chunk(Z div 100, [Z rem 100 | Parts]) 1> end. #Fun 2> Pack = fun 2> Stuff([], Bin) -> Bin; 2> Stuff([H | T], Bin) -> Stuff(T, <>) 2> end. #Fun 3> N = 1450000001. 1450000001 4> Splitted = Split(N, []). [14,50,0,0,1] 5> Packed = Pack(Splitted, <<>>). <<14,50,0,0,1>> BUT THAT IS A LITTLE BIT RIDICULOUS. On your sensor side you should have this "number" as a byte array of some sort. Figure out what the field sizes are in bits or bytes and interpret it accordingly. That's a much more sane way to do business. -Craig From dm.klionsky@REDACTED Wed Jun 20 11:33:53 2018 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 20 Jun 2018 12:33:53 +0300 Subject: [erlang-questions] Erlang first appeared year Message-ID: <60166aae-2ecd-a422-175b-39cf6a18a8b9@gmail.com> Hi all, Wikipedia https://en.wikipedia.org/wiki/Erlang_(programming_language) states that Erlang first appeared in 1986, which makes it "old" comparing to Java (1995) and C# (2000). The other day a manager said that some C++ devs mentioned that Erlang is "an old language". I replied that C++, which first appeared in 1985, is even older. Today I was reading http://blog.erlang.org/beam-compiler-history/ and realized that the year 1986 is misleading. It seems to me, that both Java and C++ have their first public releases as first appeared years and NOT when their design was started. They both have history sections mentioning that work on them was started long before. Shouldn't we consider OTP R1B in 1996 to be the first release? This will make Erlang is younger than Java! I don't propose to cheat, I propose to play the fair game. Thank you From thomas.elsgaard@REDACTED Wed Jun 20 11:42:11 2018 From: thomas.elsgaard@REDACTED (Thomas Elsgaard) Date: Wed, 20 Jun 2018 11:42:11 +0200 Subject: [erlang-questions] Erlang first appeared year In-Reply-To: <60166aae-2ecd-a422-175b-39cf6a18a8b9@gmail.com> References: <60166aae-2ecd-a422-175b-39cf6a18a8b9@gmail.com> Message-ID: Younger might not always be better... Thomas ons. 20. jun. 2018 kl. 11.34 skrev Dmitry Klionsky : > Hi all, > > Wikipedia https://en.wikipedia.org/wiki/Erlang_(programming_language) > states that Erlang > first appeared in 1986, which makes it "old" comparing to Java (1995) > and C# (2000). > The other day a manager said that some C++ devs mentioned that Erlang is > "an old language". > I replied that C++, which first appeared in 1985, is even older. > > Today I was reading http://blog.erlang.org/beam-compiler-history/ and > realized that the year > 1986 is misleading. > > It seems to me, that both Java and C++ have their first public releases > as first appeared years > and NOT when their design was started. They both have history sections > mentioning that work on > them was started long before. > > Shouldn't we consider OTP R1B in 1996 to be the first release? > This will make Erlang is younger than Java! > > I don't propose to cheat, I propose to play the fair game. > > Thank you > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Wed Jun 20 12:07:40 2018 From: bob@REDACTED (Bob Cowdery) Date: Wed, 20 Jun 2018 11:07:40 +0100 Subject: [erlang-questions] Binaries In-Reply-To: <2815626.v6yVoIg6k3@burrito> References: <148bb878-5b5a-5700-a9bd-8250d7af8044@bobcowdery.plus.com> <2815626.v6yVoIg6k3@burrito> Message-ID: <01744eca-3405-49b2-63fc-ff889829ee71@bobcowdery.plus.com> I seem to have arrived at something that works. This is only a hack for testing. I picked hexstr_to_bin up to do the conversion. I had trouble with some of the other methods on offer but that's probably me. The target docs I have say nothing about flow control so I set it to all available options but gen_serial:bsend/2 was more often than not hanging although the command was executed. I have used gen_serial:asend/2 which seems to work reliably so don't know what the issue is. I'm sure the code can be reduced quite a lot. hexstr_to_bin(S) -> ? hexstr_to_bin(S, []). hexstr_to_bin([], Acc) -> ? list_to_binary(lists:reverse(Acc)); hexstr_to_bin([X,Y|T], Acc) -> ? {ok, [V], []} = io_lib:fread("~16u", [X,Y]), ? hexstr_to_bin(T, [V | Acc]); hexstr_to_bin([X|T], Acc) -> ? {ok, [V], []} = io_lib:fread("~16u", lists:flatten([X,"0"])), hexstr_to_bin(T, [V | Acc]). I ended up with this code: open(Device) -> ??? case gen_serial:open(Device, [{baud, 9600},{stop_bits, 2},{flow_control, none}]) of ??? ??? {ok, Port} -> ??? ??? ??? Port; ??? ??? {error, Reason} -> ??? ??? ??? io:format("Error opening serial port ~w~n", Reason), ??? ??? ??? {error, Reason} ??? end. set_freq(Port, Freq) -> ??? % The command is 4 hex freq bytes followed by 0x01 (set_freq command) ??? SFreq = integer_to_list(floor(Freq*100000)), ??? io:format("~w~n", [SFreq]), ??? SPadded = string:right(SFreq, 8, $0), ??? io:format("~w~n", [SPadded]), ??? BinFreq = hex:hexstr_to_bin(SPadded), ??? BinCmd = <<16#01>>, ??? Bin = << BinFreq/binary, BinCmd/binary >>, ??? io:format("~w~n", [Bin]), ??? gen_serial:asend(Port, Bin), ??? io:format("Done send~n"), ??? gen_serial:recv(Port, 5, 2000), ??? io:format("Done response~n"). C:\Users\User>erl Running .erlang Eshell V9.3? (abort with ^G) 1> Port = cat_client:open(3). {gen_serial,<0.59.0>,#Port<0.519>} 2> cat_client:set_freq(Port, 3.8). [51,56,48,48,48,48] [48,48,51,56,48,48,48,48] <<0,56,0,0,1>> Done send Done response ok On 6/20/2018 2:21 AM, zxq9 wrote: > On 2018?6?19? ??? 16:32:49 Bob Cowdery wrote: >> If I have a number, say 1450000001 and I want to represent that as a >> binary in the form >> >> <<16#14,16#50,16#00,16#00,16#01>> what's the best way. >> >> I'm not sure what list_to_binary(integer_to_list(1450000001)) which >> prints as <<"1450000001">> produces but I guess its 10 bytes not 5. > Binary syntax can be a bit of a wonderland at first. > > The form you present above works just fine. Keep in mind that it yields the following: > > 1> <<16#14,16#50,16#00,16#00,16#01>>. > <<20,80,0,0,1>> > > If we want to encode it using your original decimal representation (or as an integer programatically generated as a variable), and pack it into the same 40-bit space (5 8-bit bytes): > > 2> <<1450000001:40>>. > <<0,86,109,62,129>> > > What, what? These aren't the same! > > There are two reasons: > - The hexadecimal modifier 16# means 10#14 =/= 16#14 > - Your version splits this integer value up into decimal-determined fields > > It appears that you don't actually want to encode the number 1450000001, but rather want five 8-bit fields carrying the values 14, 50, 00, 00, and 01, in that order. Is this correct? > > If you're starting with an integer that is *actually* 1,450,000,001 and you want to break it up by its decimal value then you'll want to div/rem it down by 100s to break it into chunks and then encode each chunk: > > 1> Split = fun > 1> Chunk(0, Parts) -> Parts; > 1> Chunk(Z, Parts) -> Chunk(Z div 100, [Z rem 100 | Parts]) > 1> end. > #Fun > 2> Pack = fun > 2> Stuff([], Bin) -> Bin; > 2> Stuff([H | T], Bin) -> Stuff(T, <>) > 2> end. > #Fun > 3> N = 1450000001. > 1450000001 > 4> Splitted = Split(N, []). > [14,50,0,0,1] > 5> Packed = Pack(Splitted, <<>>). > <<14,50,0,0,1>> > > BUT THAT IS A LITTLE BIT RIDICULOUS. > > On your sensor side you should have this "number" as a byte array of some sort. Figure out what the field sizes are in bits or bytes and interpret it accordingly. That's a much more sane way to do business. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From greim@REDACTED Wed Jun 20 12:35:18 2018 From: greim@REDACTED (greim) Date: Wed, 20 Jun 2018 12:35:18 +0200 Subject: [erlang-questions] Erlang first appeared year In-Reply-To: References: <60166aae-2ecd-a422-175b-39cf6a18a8b9@gmail.com> Message-ID: <8ffa7e69-73db-bafe-4861-c25d2e744b55@schleibinger.com> Am 20.06.2018 um 11:42 schrieb Thomas Elsgaard: > Younger might not always be better... ...I absolute agree. Or lets say "surviving of the fittest" if we define the development of programming languages as an evolutionary process. worms and jellyfishes, sharks and crocodiles are still alive...dinosaurs aren't! Markus Greim > > Thomas > ons. 20. jun. 2018 kl. 11.34 skrev Dmitry Klionsky > >: > > Hi all, > > Wikipedia https://en.wikipedia.org/wiki/Erlang_(programming_language) > states that Erlang > first appeared in 1986, which makes it "old" comparing to Java (1995) > and C# (2000). > The other day a manager said that some C++ devs mentioned that > Erlang is > "an old language". > I replied that C++, which first appeared in 1985, is even older. > > Today I was reading http://blog.erlang.org/beam-compiler-history/ and > realized that the year > 1986 is misleading. > > It seems to me, that both Java and C++ have their first public releases > as first appeared years > and NOT when their design was started. They both have history sections > mentioning that work on > them was started long before. > > Shouldn't we consider OTP R1B in 1996 to be the first release? > This will make Erlang is younger than Java! > > I don't propose to cheat, I propose to play the fair game. > > Thank you > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From hugo@REDACTED Wed Jun 20 12:39:23 2018 From: hugo@REDACTED (Hugo Mills) Date: Wed, 20 Jun 2018 10:39:23 +0000 Subject: [erlang-questions] Erlang first appeared year In-Reply-To: <8ffa7e69-73db-bafe-4861-c25d2e744b55@schleibinger.com> References: <60166aae-2ecd-a422-175b-39cf6a18a8b9@gmail.com> <8ffa7e69-73db-bafe-4861-c25d2e744b55@schleibinger.com> Message-ID: <20180620103923.GE28972@carfax.org.uk> On Wed, Jun 20, 2018 at 12:35:18PM +0200, greim wrote: > Am 20.06.2018 um 11:42 schrieb Thomas Elsgaard: > >Younger might not always be better... > > ...I absolute agree. > > Or lets say "surviving of the fittest" if we define the development > of programming languages as an evolutionary process. > > worms and jellyfishes, sharks and crocodiles are still > alive...dinosaurs aren't! Well, we call them "birds" nowadays. Hugo. > Markus Greim > > > > > > > >Thomas > >ons. 20. jun. 2018 kl. 11.34 skrev Dmitry Klionsky > >>: > > > > Hi all, > > > > Wikipedia https://en.wikipedia.org/wiki/Erlang_(programming_language) > > states that Erlang > > first appeared in 1986, which makes it "old" comparing to Java (1995) > > and C# (2000). > > The other day a manager said that some C++ devs mentioned that > > Erlang is > > "an old language". > > I replied that C++, which first appeared in 1985, is even older. > > > > Today I was reading http://blog.erlang.org/beam-compiler-history/ and > > realized that the year > > 1986 is misleading. > > > > It seems to me, that both Java and C++ have their first public releases > > as first appeared years > > and NOT when their design was started. They both have history sections > > mentioning that work on > > them was started long before. > > > > Shouldn't we consider OTP R1B in 1996 to be the first release? > > This will make Erlang is younger than Java! > > > > I don't propose to cheat, I propose to play the fair game. > > > > Thank you > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > >_______________________________________________ > >erlang-questions mailing list > >erlang-questions@REDACTED > >http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Hugo Mills | Alert status mauve ocelot: Slight chance of hugo@REDACTED carfax.org.uk | brimstone. Be prepared to make a nice cup of tea. http://carfax.org.uk/ | PGP: E2AB1DE4 | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From carlsson.richard@REDACTED Wed Jun 20 12:41:18 2018 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 20 Jun 2018 12:41:18 +0200 Subject: [erlang-questions] Erlang first appeared year In-Reply-To: <60166aae-2ecd-a422-175b-39cf6a18a8b9@gmail.com> References: <60166aae-2ecd-a422-175b-39cf6a18a8b9@gmail.com> Message-ID: Yes, 1986 is a misleading date, because that is only when experimentation started. A better date would be late 1988 or 1989-1990, or even 1992-1993 depending on your point of reference. Quoting from Joe Armstrong's paper 'A History of Erlang': - "By the end of 1988, most of the ideas in Erlang had stabilized." - "1989 also provided us with one of our first opportunities to present Erlang to the world outside Ericsson. This was when we presented a paper at the SETSS conference in Bournemouth." - "Later in the year, in December 1989, this resulted in Bjarne, Mike, Robert and me visiting Bellcore, where we gave our first ever external Erlang lecture." But it was still only a very bare-bones Erlang back then, and it was implemented on top of Prolog, and had no real commercial use yet, only experimental within Ericsson. The first efficient implementation, the JAM abstract machine, was created in 1990. - "One of the high points of 1990 was ISS?90 (International Switching Symposium), held in Stockholm. ISS?90 was the first occasion where we actively tried to market Erlang." - "In 1992, we got permission to publish a book and it was decided to commercialize Erlang. A contract was signed with Prentice Hall and the first Erlang book appeared in the bookshops in May 1993." So we can definitely say that by 1993, Erlang was a full, stable real world language with a reliable and efficient implementation and publicly available documentation. For comparison, C++ experimentation started in 1979 and the first book was published in 1985, while Perl was developed 1987-1991, Python started in 1989, Ruby was created in 1995-1996, and Java was developed in 1991-1996. Erlang is not "old". /Richard 2018-06-20 11:33 GMT+02:00 Dmitry Klionsky : > Hi all, > > Wikipedia https://en.wikipedia.org/wiki/Erlang_(programming_language) > states that Erlang > first appeared in 1986, which makes it "old" comparing to Java (1995) and > C# (2000). > The other day a manager said that some C++ devs mentioned that Erlang is > "an old language". > I replied that C++, which first appeared in 1985, is even older. > > Today I was reading http://blog.erlang.org/beam-compiler-history/ and > realized that the year > 1986 is misleading. > > It seems to me, that both Java and C++ have their first public releases as > first appeared years > and NOT when their design was started. They both have history sections > mentioning that work on > them was started long before. > > Shouldn't we consider OTP R1B in 1996 to be the first release? > This will make Erlang is younger than Java! > > I don't propose to cheat, I propose to play the fair game. > > Thank you > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonas.falkevik@REDACTED Wed Jun 20 13:56:27 2018 From: jonas.falkevik@REDACTED (Jonas Falkevik) Date: Wed, 20 Jun 2018 13:56:27 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On Mon, Jun 18, 2018 at 3:57 PM, Lukas Larsson wrote: > > >>> Something like https://github.com/falkevik/ot >> p/commit/444fb00ff2a9d1f40a8c66f48bea1cf3f07ca86c ? >> > > I was thinking something more like this: https://github.com/garazdawi/o > tp/tree/lukas/erts/realloc_thr_pref_shrink_th > > This makes it so that the absolute and relative single block shrink > thesholds are respected for reallocs made on a remote scheduler. This > should solve the problem that you have found also, through as I still > haven't reproduced it I can't test that it actually solves it. > > Are you adding the multicast network to the loop back interface? Using some other interface that does not allow multicast traffic? I have been able to reproduce it in macOS Sierra (Darwin Kernel Version 16.7.0: Fri Apr 27 17:59:46 PDT 2018; root:xnu-3789.73.13~1/RELEASE_X86_64 x86_64) and also on Ubuntu Linux 16.04 LTS with kernel 4.4.0-112-generic. Danil Zagoskin, do you have time to try and see if there is any change in the behaviour for you with a patched system? With the bugfix, the beam is spending most of its time in sched_spin_wait on linux and sched_yield on macOS. Stats below are from OTP 21 and then OTP21 with bugfix on linux Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] Eshell V10.0 (abort with ^G) 1> udptest:start_sender({239,9,9,9}, 3999). <0.78.0> 2> [udptest:start_reader({239,9,9,9}, 3999) || _ <- lists:seq(1, 40)]. [<0.80.0>,<0.81.0>,<0.82.0>,<0.83.0>,<0.84.0>,<0.85.0>, <0.86.0>,<0.87.0>,<0.88.0>,<0.89.0>,<0.90.0>,<0.91.0>, <0.92.0>,<0.93.0>,<0.94.0>,<0.95.0>,<0.96.0>,<0.97.0>, <0.98.0>,<0.99.0>,<0.100.0>,<0.101.0>,<0.102.0>,<0.103.0>, <0.104.0>,<0.105.0>,<0.106.0>,<0.107.0>,<0.108.0>|...] 3> msacc:start(10000), msacc:print(). Average thread real-time : 10002138 us Accumulated system run-time : 62576106 us Average scheduler run-time : 7775309 us Thread aux check_io emulator gc other port sleep Stats per thread: async( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% aux( 1) 0.06% 0.96% 0.00% 0.00% 0.07% 0.00% 98.90% dirty_cpu_( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 9) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s(10) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% poll( 0) 0.00% 2.63% 0.00% 0.00% 0.00% 0.00% 97.37% scheduler( 1) 0.82% 0.00% 0.52% 0.35% 7.33% 67.91% 23.08% scheduler( 2) 0.76% 0.00% 0.79% 0.34% 7.30% 68.61% 22.21% scheduler( 3) 0.74% 0.00% 0.83% 0.33% 7.17% 68.54% 22.38% scheduler( 4) 0.76% 0.00% 0.72% 0.34% 7.38% 68.84% 21.96% scheduler( 5) 0.79% 0.00% 0.82% 0.34% 7.21% 68.94% 21.90% scheduler( 6) 0.80% 0.00% 0.74% 0.31% 7.15% 68.56% 22.44% scheduler( 7) 0.87% 0.00% 0.72% 0.37% 7.24% 68.43% 22.36% scheduler( 8) 0.77% 0.00% 0.68% 0.35% 7.41% 69.02% 21.77% Stats per type: async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% aux 0.06% 0.96% 0.00% 0.00% 0.07% 0.00% 98.90% dirty_cpu_sche 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% poll 0.00% 2.63% 0.00% 0.00% 0.00% 0.00% 97.37% scheduler 0.79% 0.00% 0.73% 0.34% 7.27% 68.61% 22.26% ok 4> $ ../src/udp_performace_bug_fix/bin/erl Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-9.3.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] Eshell V9.3.2 (abort with ^G) 1> udptest:start_sender({239,9,9,9}, 3999). <0.78.0> 2> [udptest:start_reader({239,9,9,9}, 3999) || _ <- lists:seq(1, 40)]. [<0.80.0>,<0.81.0>,<0.82.0>,<0.83.0>,<0.84.0>,<0.85.0>, <0.86.0>,<0.87.0>,<0.88.0>,<0.89.0>,<0.90.0>,<0.91.0>, <0.92.0>,<0.93.0>,<0.94.0>,<0.95.0>,<0.96.0>,<0.97.0>, <0.98.0>,<0.99.0>,<0.100.0>,<0.101.0>,<0.102.0>,<0.103.0>, <0.104.0>,<0.105.0>,<0.106.0>,<0.107.0>,<0.108.0>|...] 3> msacc:start(10000), msacc:print(). Average thread real-time : 10000416 us Accumulated system run-time : 19541445 us Average scheduler run-time : 2279822 us Thread aux check_io emulator gc other port sleep Stats per thread: async( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% aux( 1) 0.00% 10.72% 0.00% 0.00% 0.00% 0.00% 89.28% dirty_cpu_( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_cpu_( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s( 9) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_s(10) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% poll( 0) 0.00% 2.31% 0.00% 0.00% 0.00% 0.00% 97.69% scheduler( 1) 0.46% 0.00% 0.43% 0.14% 17.57% 4.07% 77.33% scheduler( 2) 0.50% 0.00% 0.44% 0.15% 17.13% 4.21% 77.57% scheduler( 3) 0.49% 0.00% 0.47% 0.16% 17.07% 4.35% 77.46% scheduler( 4) 0.45% 0.00% 0.42% 0.14% 16.25% 4.02% 78.72% scheduler( 5) 0.48% 0.00% 0.42% 0.14% 17.04% 4.04% 77.88% scheduler( 6) 0.46% 0.00% 0.39% 0.14% 16.57% 3.77% 78.67% scheduler( 7) 0.53% 0.00% 1.88% 0.27% 17.60% 8.22% 71.50% scheduler( 8) 0.47% 0.00% 0.42% 0.15% 16.60% 3.86% 78.50% Stats per type: async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% aux 0.00% 10.72% 0.00% 0.00% 0.00% 0.00% 89.28% dirty_cpu_sche 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% poll 0.00% 2.31% 0.00% 0.00% 0.00% 0.00% 97.69% scheduler 0.48% 0.00% 0.61% 0.16% 16.98% 4.57% 77.20% ok Jonas -------------- next part -------------- An HTML attachment was scrubbed... URL: From Catenacci@REDACTED Wed Jun 20 14:46:54 2018 From: Catenacci@REDACTED (Onorio Catenacci) Date: Wed, 20 Jun 2018 08:46:54 -0400 Subject: [erlang-questions] [ANN] Chocolatey Nuget Package For Erlang Updated To Erlang 21 In-Reply-To: References: Message-ID: Hi all, I've updated the Chocolatey Nuget Package for Erlang to version 21. Just wanted to make sure people are aware. -- Onorio Catenacci http://onor.io http://www.google.com/+OnorioCatenacci -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Jun 20 15:03:51 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 20 Jun 2018 16:03:51 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: Wow! This is a serious change. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Wed Jun 20 15:30:20 2018 From: g@REDACTED (Guilherme Andrade) Date: Wed, 20 Jun 2018 14:30:20 +0100 Subject: [erlang-questions] enif_send() and overrun heap Message-ID: Hello list, I'm bumping into a weird issue in OTP 20.3 (macOS) whereby calling enif_send() a few dozen times a second from a dirty scheduler (CPU bound), with msg_env=NULL, results in heap overrun. These is the flow that sooner or later results in heap overrun: 1) Single Erlang process makes a specific NIF call ~25 times per second (dirty scheduler, CPU bound) 2) This call will receive a few network packets (non-blocking) 3) Each of these packets gets wrapped in a tuple (allocated in process env) 4) For each of these wrapped packets, a lookup is made in a map, passed as a NIF argument, for a process dedicated to handling this particular packet 5.a) If the lookup succeeds, enif_send() is called to dispatch the wrapped packet to said dedicated process (with msg_env=NULL) - this is what happens to most packets 5.b) If the lookup fails, the wrapped packet is accumulated and later returned to the NIF caller Now, when total packets per second increase to a few dozen, sooner or later (sometimes as soon as after ~10 seconds) the VM stops abruptly with this error message: > hend=0x0000000013655fa0 > stop=0x0000000013655ef8 > htop=0x00000000136562c8 > heap=0x0000000013652db0 > beam/erl_gc.c, line 708: <0.506.0>: Overrun stack and heap (The pid mentioned above corresponds to the NIF caller.) I tried three things (independently) that prevent the overrun from happening under this load: A) Increasing the NIF caller heap size from the default (233 words) to 23300 words B) Not running the NIF under a dirty scheduler C) Not calling enif_send Any ideas on why the overrun happens? Am I missing some very obvious transgression in the way enif_send() or dirty schedulers are supposed to be used? -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Wed Jun 20 15:42:26 2018 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Wed, 20 Jun 2018 08:42:26 -0500 Subject: [erlang-questions] enif_send() and overrun heap In-Reply-To: References: Message-ID: I wonder if running under a dirty scheduler counts as "running from a created thread" in which case the first parameter to enif_send() should be NULL. On Wed, 20 Jun 2018 at 08:30 Guilherme Andrade wrote: > Hello list, > > I'm bumping into a weird issue in OTP 20.3 (macOS) whereby calling > enif_send() a few dozen times a second from a dirty scheduler (CPU bound), > with msg_env=NULL, results in heap overrun. > > These is the flow that sooner or later results in heap overrun: > > 1) Single Erlang process makes a specific NIF call ~25 times per second > (dirty scheduler, CPU bound) > 2) This call will receive a few network packets (non-blocking) > 3) Each of these packets gets wrapped in a tuple (allocated in process env) > 4) For each of these wrapped packets, a lookup is made in a map, passed as > a NIF argument, for a process dedicated to handling this particular packet > 5.a) If the lookup succeeds, enif_send() is called to dispatch the wrapped > packet to said dedicated process (with msg_env=NULL) - this is what happens > to most packets > 5.b) If the lookup fails, the wrapped packet is accumulated and later > returned to the NIF caller > > Now, when total packets per second increase to a few dozen, sooner or > later (sometimes as soon as after ~10 seconds) the VM stops abruptly with > this error message: > > > hend=0x0000000013655fa0 > > stop=0x0000000013655ef8 > > htop=0x00000000136562c8 > > heap=0x0000000013652db0 > > beam/erl_gc.c, line 708: <0.506.0>: Overrun stack and heap > > (The pid mentioned above corresponds to the NIF caller.) > > I tried three things (independently) that prevent the overrun from > happening under this load: > A) Increasing the NIF caller heap size from the default (233 words) to > 23300 words > B) Not running the NIF under a dirty scheduler > C) Not calling enif_send > > Any ideas on why the overrun happens? Am I missing some very obvious > transgression in the way enif_send() or dirty schedulers are supposed to be > used? > > > -- > Guilherme > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.whealy@REDACTED Wed Jun 20 15:43:31 2018 From: chris.whealy@REDACTED (Whealy, Chris) Date: Wed, 20 Jun 2018 13:43:31 +0000 Subject: [erlang-questions] Weird dependency: Failed connection to MongoDB instance causes call to ets:lookup/2 to fail with badarg Message-ID: Hi All This is more of an FYI... I have a Cowboy application that does the following during start up: 1. Start iBrowse 2. Use iBrowse to make an HTTP request to download a text file containing country information 3. Parse text file to create a list of countries 4. Start Cowboy and define routes 5. Start supervisor passing it the list of countries 6. Supervisor waits for HTTP requests? This all works nicely. I?m now extending the functionality to include a MongoDB. At first, I modified the startup sequence to this: 1. Connect to MongoDB instance and get pid of db server 2. Start iBrowse 3. Use iBrowse to make an HTTP request to download a text file containing country information 4. Parse text file to create a list of countries 5. Start Cowboy and define routes 6. Start supervisor passing it the list of countries and MongoDB pid 7. Supervisor waits for HTTP requests? If the MongoDB instance is up, then no problem. However, if the MongoDB instance is down, then step 1 returns the expected {error, econnrefused} when calling mc_worker_api:connect/1 and the db server pid defaults to undefined. Again, this is all as expected... But the weird thing is that if the connection to MongoDB fails, then step 3 subsequently fails when iBrowse tries to download the text file. The specific failure is badarg when calling ets:lookup/2 in line 345 of ibrowse.erl. This doesn?t make much sense to me at all because the two libraries should be unrelated. I can make the problem go away by moving the code that connects to MongoDB to be after the iBrowse request, but to me, the weirdness of this error highlights that there might be some underlying problem with the MongoDB library and ets. Regards Chris Whealy SAP Cloud Platform | Strategy & Product Management | Team SAP UK Ltd, Clockhouse Place, Bedfont Rd, Feltham, Middx, TW14 8HA, England M +44 (0)7808 575377 Find out more on the Strategy & Product Management Wiki page (SAP internal) Follow our latest activities in SAP CP User Community Jam Group Please consider the impact on the environment before printing this e-mail. Twitter: @LogaRhythm "The voice of ignorance speaks loud and long, but the words of the wise are quiet and few" Ancient Proverb -------------- next part -------------- An HTML attachment was scrubbed... URL: From greim@REDACTED Wed Jun 20 15:56:53 2018 From: greim@REDACTED (greim) Date: Wed, 20 Jun 2018 15:56:53 +0200 Subject: [erlang-questions] Erlang first appeared year In-Reply-To: References: <60166aae-2ecd-a422-175b-39cf6a18a8b9@gmail.com> <8ffa7e69-73db-bafe-4861-c25d2e744b55@schleibinger.com> Message-ID: Am 20.06.2018 um 12:41 schrieb Daron Ryan: > Dinosaurs are extinct? Pascal / Delphi is still alive. Delphi is for hipsters....I still use Borland Pascal 7.01 nearly daily for a now 24 years running embedded project. A real world application 24/7/365...and even ERLANG (since 2 years). Maybe it requires > something like a rock from outer space to hit the gulf of Mexico to draw > the line (sorry I don't know whether the right word is asteroid or > something else so I played it safe and said rock) Who ceres about dying? Have you ever heard about reincarnation? MISRA-C ......Pascal with curly brackets, how awful ;-) but now we are running out of topic here.. Markus https://en.wikipedia.org/wiki/MISRA_C > > On 20 Jun 2018 8:35 PM, "greim" > wrote: > > Am 20.06.2018 um 11:42 schrieb Thomas Elsgaard: > > Younger might not always be better... > > > ...I absolute agree. > > Or lets say "surviving of the fittest" if we define the development > of programming languages as an evolutionary process. > > worms and jellyfishes, sharks and crocodiles are still > alive...dinosaurs aren't! > > Markus Greim > > > > > > Thomas > ons. 20. jun. 2018 kl. 11.34 skrev Dmitry Klionsky > > >>: > > ? ? Hi all, > > ? ? Wikipedia > https://en.wikipedia.org/wiki/Erlang_(programming_language) > > ? ? states that Erlang > ? ? first appeared in 1986, which makes it "old" comparing to > Java (1995) > ? ? and C# (2000). > ? ? The other day a manager said that some C++ devs mentioned that > ? ? Erlang is > ? ? "an old language". > ? ? I replied that C++, which first appeared in 1985, is even > older. > > ? ? Today I was reading > http://blog.erlang.org/beam-compiler-history/ > and > ? ? realized that the year > ? ? 1986 is misleading. > > ? ? It seems to me, that both Java and C++ have their first > public releases > ? ? as first appeared years > ? ? and NOT when their design was started. They both have > history sections > ? ? mentioning that work on > ? ? them was started long before. > > ? ? Shouldn't we consider OTP R1B in 1996 to be the first release? > ? ? This will make Erlang is younger than Java! > > ? ? I don't propose to cheat, I propose to play the fair game. > > ? ? Thank you > > ? ? _______________________________________________ > ? ? erlang-questions mailing list > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From g@REDACTED Wed Jun 20 16:18:05 2018 From: g@REDACTED (Guilherme Andrade) Date: Wed, 20 Jun 2018 15:18:05 +0100 Subject: [erlang-questions] enif_send() and overrun heap In-Reply-To: References: Message-ID: On Wed, 20 Jun 2018, 14:42 Daniel Goertzen, wrote: > I wonder if running under a dirty scheduler counts as "running from a > created thread" in which case the first parameter to enif_send() should be > NULL. > Good point! That seemed to do it, thanks. May someone from OTP team confirm dirty schedulers count as created threads for enif_send? (It makes perfect sense they would, but it's best to check.) If confirmed, maybe I'll do a PR to make documentation more explicit about this corner case. > On Wed, 20 Jun 2018 at 08:30 Guilherme Andrade wrote: > >> Hello list, >> >> I'm bumping into a weird issue in OTP 20.3 (macOS) whereby calling >> enif_send() a few dozen times a second from a dirty scheduler (CPU bound), >> with msg_env=NULL, results in heap overrun. >> >> These is the flow that sooner or later results in heap overrun: >> >> 1) Single Erlang process makes a specific NIF call ~25 times per second >> (dirty scheduler, CPU bound) >> 2) This call will receive a few network packets (non-blocking) >> 3) Each of these packets gets wrapped in a tuple (allocated in process >> env) >> 4) For each of these wrapped packets, a lookup is made in a map, passed >> as a NIF argument, for a process dedicated to handling this particular >> packet >> 5.a) If the lookup succeeds, enif_send() is called to dispatch the >> wrapped packet to said dedicated process (with msg_env=NULL) - this is what >> happens to most packets >> 5.b) If the lookup fails, the wrapped packet is accumulated and later >> returned to the NIF caller >> >> Now, when total packets per second increase to a few dozen, sooner or >> later (sometimes as soon as after ~10 seconds) the VM stops abruptly with >> this error message: >> >> > hend=0x0000000013655fa0 >> > stop=0x0000000013655ef8 >> > htop=0x00000000136562c8 >> > heap=0x0000000013652db0 >> > beam/erl_gc.c, line 708: <0.506.0>: Overrun stack and heap >> >> (The pid mentioned above corresponds to the NIF caller.) >> >> I tried three things (independently) that prevent the overrun from >> happening under this load: >> A) Increasing the NIF caller heap size from the default (233 words) to >> 23300 words >> B) Not running the NIF under a dirty scheduler >> C) Not calling enif_send >> >> Any ideas on why the overrun happens? Am I missing some very obvious >> transgression in the way enif_send() or dirty schedulers are supposed to be >> used? >> >> >> -- >> Guilherme >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Wed Jun 20 17:17:32 2018 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Wed, 20 Jun 2018 10:17:32 -0500 Subject: [erlang-questions] enif_send() and overrun heap In-Reply-To: References: Message-ID: Hmm, I think there's more to it. The source shows that dirty schedulers *are* specifically handled in enif_send(), so something else is up. On Wed, 20 Jun 2018 at 09:18 Guilherme Andrade wrote: > > On Wed, 20 Jun 2018, 14:42 Daniel Goertzen, > wrote: > >> I wonder if running under a dirty scheduler counts as "running from a >> created thread" in which case the first parameter to enif_send() should be >> NULL. >> > > Good point! That seemed to do it, thanks. > > May someone from OTP team confirm dirty schedulers count as created > threads for enif_send? (It makes perfect sense they would, but it's best to > check.) > > If confirmed, maybe I'll do a PR to make documentation more explicit about > this corner case. > > > >> On Wed, 20 Jun 2018 at 08:30 Guilherme Andrade wrote: >> >>> Hello list, >>> >>> I'm bumping into a weird issue in OTP 20.3 (macOS) whereby calling >>> enif_send() a few dozen times a second from a dirty scheduler (CPU bound), >>> with msg_env=NULL, results in heap overrun. >>> >>> These is the flow that sooner or later results in heap overrun: >>> >>> 1) Single Erlang process makes a specific NIF call ~25 times per second >>> (dirty scheduler, CPU bound) >>> 2) This call will receive a few network packets (non-blocking) >>> 3) Each of these packets gets wrapped in a tuple (allocated in process >>> env) >>> 4) For each of these wrapped packets, a lookup is made in a map, passed >>> as a NIF argument, for a process dedicated to handling this particular >>> packet >>> 5.a) If the lookup succeeds, enif_send() is called to dispatch the >>> wrapped packet to said dedicated process (with msg_env=NULL) - this is what >>> happens to most packets >>> 5.b) If the lookup fails, the wrapped packet is accumulated and later >>> returned to the NIF caller >>> >>> Now, when total packets per second increase to a few dozen, sooner or >>> later (sometimes as soon as after ~10 seconds) the VM stops abruptly with >>> this error message: >>> >>> > hend=0x0000000013655fa0 >>> > stop=0x0000000013655ef8 >>> > htop=0x00000000136562c8 >>> > heap=0x0000000013652db0 >>> > beam/erl_gc.c, line 708: <0.506.0>: Overrun stack and heap >>> >>> (The pid mentioned above corresponds to the NIF caller.) >>> >>> I tried three things (independently) that prevent the overrun from >>> happening under this load: >>> A) Increasing the NIF caller heap size from the default (233 words) to >>> 23300 words >>> B) Not running the NIF under a dirty scheduler >>> C) Not calling enif_send >>> >>> Any ideas on why the overrun happens? Am I missing some very obvious >>> transgression in the way enif_send() or dirty schedulers are supposed to be >>> used? >>> >>> >>> -- >>> Guilherme >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Jun 20 19:06:21 2018 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 20 Jun 2018 19:06:21 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On Wed, Jun 20, 2018 at 1:56 PM Jonas Falkevik < jonas.falkevik@REDACTED> wrote: > > Are you adding the multicast network to the loop back interface? Using > some other interface that does not allow multicast traffic? > > I though I had, but apparently not.... I managed to reproduce the test case now. I also spent some time staring at the code in the inet_driver and realized what the problem was in there. It would seem that a performance/feature fix for SCTP in R15B inadvertently introduced this behaviour for UDP :( I've pushed a bug fix to that problem in the inet_driver to the same branch. In my tests the scheduler utilization goes form 71% to about 4% with both patches. With the fix in the inet driver, my allocator patch does not really do any difference for this specific testcase, but I'll keep that fix anyways as it is a "good thing". I should probably also add a benchmark for this so that it does not happen again.... I'd also like to add that changing the test-case to use a smaller user-space buffer also has the same effect. So if anyone is running a performance critical UDP server that has set the "recbuf" to a high value, I really recommend that you lower the "buffer" size to something close to the max expected packet size. In the example that Danil provided I applied the following patch: diff --git a/udptest.erl b/udptest.erl index 16a1798..4edeef0 100644 --- a/udptest.erl +++ b/udptest.erl @@ -33,7 +33,9 @@ send_packet(ID, S, Addr, Port) -> start_reader(Addr, Port) -> GwIP = {0,0,0,0}, % {127,0,0,1}, - Common = [binary,{reuseaddr,true},{recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], + Common = [binary,{reuseaddr,true}, + {buffer,1500}, %% 1500 is just an example value, don't just copy this. You need to know what you max UDP packet size will be. + {recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], Multicast = [{multicast_ttl,4},{multicast_loop,true},{ip,Addr},{add_membership,{Addr,GwIP}}], Options = Common ++ Multicast, spawn(fun() -> run_reader(Port, Options) end). and the scheduler utilization dropped to about 4% there as well. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Wed Jun 20 19:39:04 2018 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Wed, 20 Jun 2018 17:39:04 +0000 Subject: [erlang-questions] enif_send() and overrun heap In-Reply-To: References: Message-ID: <1529516344.30324.29.camel@ericsson.com> Yes, a "created thread" refers to a thread that was not created by the VM itself. I suggest you (and everybody during NIF development) to run on a debug-VM. It's slower but it catches a lot of faults earlier and produces a nicer core dump. It will catch enif_* API violations such as building tuples/lists/maps with terms from different environments. # Build in source tree with cd $ERL_TOP/erts/emulator make debug # Run in source tree with $ERL_TOP/bin/cerl -debug ? ? ?# yes, cerl with a 'c' or if OTP-20 or newer copy $ERL_TOP/bin//beam.debug.smp and erl_child_setup.debug to same install dir as beam.smp and run with erl -emu_type debug /Sverker, Erlang/OTP Ericsson On ons, 2018-06-20 at 10:17 -0500, Daniel Goertzen wrote: > Hmm, I think there's more to it.? The source shows that dirty schedulers *are* > specifically handled in enif_send(), so something else is up. > > > > On Wed, 20 Jun 2018 at 09:18 Guilherme Andrade wrote: > > > > On Wed, 20 Jun 2018, 14:42 Daniel Goertzen, > > wrote: > > > I wonder if running under a dirty scheduler counts as "running from a > > > created thread" in which case the first parameter to enif_send() should be > > > NULL. > > > > > Good point! That seemed to do it, thanks. > > > > May someone from OTP? team confirm dirty schedulers count as created threads > > for enif_send? (It makes perfect sense they would, but it's best to check.) > > > > If confirmed, maybe I'll do a PR to make documentation more explicit about > > this corner case. > > > > > > > > > > On Wed, 20 Jun 2018 at 08:30 Guilherme Andrade wrote: > > > > Hello list, > > > > > > > > I'm bumping into a weird issue in OTP 20.3 (macOS) whereby calling > > > > enif_send() a few dozen times a second from a dirty scheduler (CPU > > > > bound), with msg_env=NULL, results in heap overrun. > > > > > > > > These is the flow that sooner or later results in heap overrun: > > > > > > > > 1) Single Erlang process makes a specific NIF call ~25 times per second > > > > (dirty scheduler, CPU bound) > > > > 2) This call will receive a few network packets (non-blocking) > > > > 3) Each of these packets gets wrapped in a tuple (allocated in process > > > > env) > > > > 4) For each of these wrapped packets, a lookup is made in a map, passed > > > > as a NIF argument, for a process dedicated to handling this particular > > > > packet > > > > 5.a) If the lookup succeeds, enif_send() is called to dispatch the > > > > wrapped packet to said dedicated process (with msg_env=NULL) - this is > > > > what happens to most packets > > > > 5.b) If the lookup fails, the wrapped packet is accumulated and later > > > > returned to the NIF caller > > > > > > > > Now, when total packets per second increase to a few dozen, sooner or > > > > later (sometimes as soon as after ~10 seconds) the VM stops abruptly > > > > with this error message: > > > > > > > > > hend=0x0000000013655fa0 > > > > > stop=0x0000000013655ef8 > > > > > htop=0x00000000136562c8 > > > > > heap=0x0000000013652db0 > > > > > beam/erl_gc.c, line 708: <0.506.0>: Overrun stack and heap > > > > > > > > (The pid mentioned above corresponds to the NIF caller.) > > > > > > > > I tried three things (independently) that prevent the overrun from > > > > happening under this load: > > > > A) Increasing the NIF caller heap size from the default (233 words) to > > > > 23300 words > > > > B) Not running the NIF under a dirty scheduler > > > > C) Not calling enif_send > > > > > > > > Any ideas on why the overrun happens? Am I missing some very obvious > > > > transgression in the way enif_send() or dirty schedulers are supposed to > > > > be used? > > > > > > > > > > > > _______________________________________________ > > > > erlang-questions mailing list > > > > erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > From rich.neswold@REDACTED Wed Jun 20 22:00:11 2018 From: rich.neswold@REDACTED (Rich Neswold) Date: Wed, 20 Jun 2018 15:00:11 -0500 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On Wed, Jun 20, 2018 at 12:07 PM Lukas Larsson wrote: > So if anyone is running a performance critical UDP server that has set the > "recbuf" to a high value, I really recommend that you lower the "buffer" > size to something close to the max expected packet size. > That's interesting because the official documentation says: "It is recommended to have val(buffer) >= max(val(sndbuf),val(recbuf)) to avoid performance issues because of unnecessary copying." Maybe the documentation is wrong because that doesn't make much sense; sndbuf and recbuf are sent to the kernel and are supposed to be bigger than the user's buffer. -- Rich -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Wed Jun 20 22:58:44 2018 From: bob@REDACTED (Bob Cowdery) Date: Wed, 20 Jun 2018 21:58:44 +0100 Subject: [erlang-questions] Help on priv_dir please In-Reply-To: References: <98afab68-9cde-88ab-09a7-c1d308887600@bobcowdery.plus.com> Message-ID: <921384ac-9656-6be0-0a1d-049127d51418@bobcowdery.plus.com> This code:priv_dir/1 has me completely baffled. The code for gen_serial will only resolve the directory if I leave it in its original directory and add the ebin to the code path. If I move the files to my own tree it wont give me anything except {error, bad_name}. I've put in the correct directory structure according to the docs, added an Application.app file and tried various names in the call including ?MODULE, the file I am running and the app name. I also added the ebin path to ERL_LIBS to no avail. There is also some weird thing going on with the paths. Even if I remove the gen_serial original code path from .erlang and then build it from some random directory it still builds that one even though I have a copy in my tree and the code path only points there. Is there some cache somewhere? On 6/19/2018 8:40 PM, Vance Shipley wrote: > Heh, I just posted about this in another thread. You probably have a > problem related to your installed path. It's not enough to set the > code path, you need to include the parent directory in the ERL_LIBS > environment variable. > > > On Tue, Jun 19, 2018, 01:52 Bob Cowdery > wrote: > > Hi, > > I'm trying to get gen_serial up and running. I've tracked the problem > down to a line which boils down to code:priv_dir(gen_serial). > > This is intending to get the port executable path but returns > bad_name. > Why would this call fail? > > Thanks > > BobC > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Wed Jun 20 23:22:24 2018 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 20 Jun 2018 23:22:24 +0200 Subject: [erlang-questions] Help on priv_dir please In-Reply-To: <921384ac-9656-6be0-0a1d-049127d51418@bobcowdery.plus.com> References: <98afab68-9cde-88ab-09a7-c1d308887600@bobcowdery.plus.com> <921384ac-9656-6be0-0a1d-049127d51418@bobcowdery.plus.com> Message-ID: First, if you run code:which(ModuleName) for any module of the app, it should show you a path that ends in .../APPNAME[-VSN]/ebin/ModuleName.ebin - if it doesn't, code:priv_dir(APPNAME) won't recognize the name as an app and you'll get {error, bad_name}. For example: code:which(lists) -> ".../stdlib-3.4/ebin/lists.beam", code:priv_dir(stdlib) -> ".../stdlib-3.4/priv". This should work even if the priv directory doesn't in fact exist (stdlib does not have one). Second, for ERL_LIBS to work, it should point out the directory above your set of applications, as in ".../lib", not an ebin directory or a single app directory. Directories under ERL_LIBS that contain an ebin subdirectory will be recognized as apps, and their ebins will be added to the path. Hope this helps. /Richard 2018-06-20 22:58 GMT+02:00 Bob Cowdery : > This code:priv_dir/1 has me completely baffled. The code for gen_serial > will only resolve the directory if I leave it in its original directory and > add the ebin to the code path. If I move the files to my own tree it wont > give me anything except {error, bad_name}. I've put in the correct > directory structure according to the docs, added an Application.app file > and tried various names in the call including ?MODULE, the file I am > running and the app name. I also added the ebin path to ERL_LIBS to no > avail. > > There is also some weird thing going on with the paths. Even if I remove > the gen_serial original code path from .erlang and then build it from some > random directory it still builds that one even though I have a copy in my > tree and the code path only points there. Is there some cache somewhere? > > On 6/19/2018 8:40 PM, Vance Shipley wrote: > > Heh, I just posted about this in another thread. You probably have a > problem related to your installed path. It's not enough to set the code > path, you need to include the parent directory in the ERL_LIBS environment > variable. > > > On Tue, Jun 19, 2018, 01:52 Bob Cowdery wrote: > >> Hi, >> >> I'm trying to get gen_serial up and running. I've tracked the problem >> down to a line which boils down to code:priv_dir(gen_serial). >> >> This is intending to get the port executable path but returns bad_name. >> Why would this call fail? >> >> Thanks >> >> BobC >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.geib.44@REDACTED Wed Jun 20 23:49:35 2018 From: mark.geib.44@REDACTED (Mark Geib) Date: Wed, 20 Jun 2018 15:49:35 -0600 Subject: [erlang-questions] with Erlang R21 rebar3 build fails Message-ID: <39CFC171-696D-4FCE-B760-3BF1F4AF8F57@gmail.com> Strange problem developed after upgrading to erlang R21 on macOS. When I do a rebar3 build now I get: geibs-MacBook-Pro:json2splunk geib$ rebar3 release ... ===> Compiling amqp_client ===> Compiling json2splunk ===> Running cuttlefish schema generator ===> Starting relx build process ... ===> Resolving OTP Applications from directories: /Users/geib/git/cse.dbsa.download.json2splunk/json2splunk/_build/default/lib /Users/geib/git/cse.dbsa.download.json2splunk/json2splunk/apps /usr/local/Cellar/erlang/21.0/lib/erlang/lib ===> Resolved json2splunk-1.1.2-1 ===> Dev mode enabled, release will be symlinked ===> release successfully created! ===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump ===> When submitting a bug report, please include the output of `rebar3 report "your command?` and rebar3.crashdump is: geibs-MacBook-Pro:json2splunk geib$ cat rebar3.crashdump Error: {badmatch,{error,eperm}} [{rlx_prv_overlay,write_template,3, [{file,"/home/tristan/Devel/rebar3/_build/default/lib/relx/src/rlx_prv_overlay.erl"}, {line,498}]}, {lists,map,2,[{file,"lists.erl"},{line,1239}]}, {lists,map,2,[{file,"lists.erl"},{line,1239}]}, {rlx_prv_overlay,do_overlay,3, [{file,"/home/tristan/Devel/rebar3/_build/default/lib/relx/src/rlx_prv_overlay.erl"}, {line,292}]}, {relx,run_provider,2, [{file,"/home/tristan/Devel/rebar3/_build/default/lib/relx/src/relx.erl"}, {line,308}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1263}]}, {relx,run_providers_for_actions,2, [{file,"/home/tristan/Devel/rebar3/_build/default/lib/relx/src/relx.erl"}, {line,291}]}, {relx,main,2, [{file,"/home/tristan/Devel/rebar3/_build/default/lib/relx/src/relx.erl"}, {line,65}]}] Now switching back to erlang R20 I get a successful build geibs-MacBook-Pro:json2splunk geib$ rebar3 release ... ===> Compiling recon ===> Compiling ranch ===> Compiling ranch_proxy_protocol ===> Compiling goldrush ===> Compiling lager ===> Compiling jsx ===> Compiling rabbit_common ===> Compiling amqp_client ===> Compiling json2splunk ===> Running cuttlefish schema generator ===> Starting relx build process ... ===> Resolving OTP Applications from directories: /Users/geib/git/cse.dbsa.download.json2splunk/json2splunk/_build/default/lib /Users/geib/git/cse.dbsa.download.json2splunk/json2splunk/apps /usr/local/Cellar/erlang@REDACTED/20.3.8/lib/erlang/lib ===> Resolved json2splunk-1.1.2-1 ===> Dev mode enabled, release will be symlinked ===> release successfully created! I assume since rebar3 is the same exec, that this is a potential issue with R21. Any ideas.??? Thanks, Mark. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 528 bytes Desc: Message signed with OpenPGP URL: From mononcqc@REDACTED Thu Jun 21 01:21:58 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 20 Jun 2018 19:21:58 -0400 Subject: [erlang-questions] with Erlang R21 rebar3 build fails In-Reply-To: <39CFC171-696D-4FCE-B760-3BF1F4AF8F57@gmail.com> References: <39CFC171-696D-4FCE-B760-3BF1F4AF8F57@gmail.com> Message-ID: <20180620232157.GG8817@ferdmbp.local> On 06/20, Mark Geib wrote: >and rebar3.crashdump is: > >geibs-MacBook-Pro:json2splunk geib$ cat rebar3.crashdump >Error: {badmatch,{error,eperm}} >[{rlx_prv_overlay,write_template,3, > [{file,"/home/tristan/Devel/rebar3/_build/default/lib/relx/src/rlx_prv_overlay.erl"}, > {line,498}]}, > ... > >Now switching back to erlang R20 I get a successful build > > >I assume since rebar3 is the same exec, that this is a potential issue with R21. >Any ideas.??? > See http://erlang.org/pipermail/erlang-questions/2018-June/095614.html OTP-21.0 sees the entire file drivers rewritten. The bug you have seems to have been reported to the OTP team during OTP-21-rc2. I don't know exactly what happened between OTP-21-rc2 and OTP-21.0, and if you'd need to run with a rebar3 built specifically on OTP-21.0 for things to work fine. From john.hogberg@REDACTED Thu Jun 21 09:41:47 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Thu, 21 Jun 2018 07:41:47 +0000 Subject: [erlang-questions] with Erlang R21 rebar3 build fails In-Reply-To: <20180620232157.GG8817@ferdmbp.local> References: <39CFC171-696D-4FCE-B760-3BF1F4AF8F57@gmail.com> <20180620232157.GG8817@ferdmbp.local> Message-ID: <1529566907.19459.33.camel@ericsson.com> Hi, This is caused by fixing a bug in OTP 21; file:write_file_info and friends used to report success on owner/group changes regardless of whether it worked or not. We marked this as an incompatibility in the release notes under OTP- 15118. Regards, John H?gberg On ons, 2018-06-20 at 19:21 -0400, Fred Hebert wrote: > On 06/20, Mark Geib wrote: > > > > and rebar3.crashdump is: > > > > geibs-MacBook-Pro:json2splunk geib$ cat rebar3.crashdump > > Error: {badmatch,{error,eperm}} > > [{rlx_prv_overlay,write_template,3, > > ?????????????????[{file,"/home/tristan/Devel/rebar3/_build/default/ > > lib/relx/src/rlx_prv_overlay.erl"}, > > ??????????????????{line,498}]}, > > ... > > > > Now switching back to erlang R20 I get a successful build > > > > > > I assume since rebar3 is the same exec, that this is a potential > > issue with R21. > > Any ideas.??? > > > See http://erlang.org/pipermail/erlang-questions/2018-June/095614.htm > l > > OTP-21.0 sees the entire file drivers rewritten. The bug you have > seems? > to have been reported to the OTP team during OTP-21-rc2. > > I don't know exactly what happened between OTP-21-rc2 and OTP-21.0, > and? > if you'd need to run with a rebar3 built specifically on OTP-21.0 > for? > things to work fine. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From lukas@REDACTED Thu Jun 21 09:46:09 2018 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 21 Jun 2018 09:46:09 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On Wed, Jun 20, 2018 at 10:00 PM Rich Neswold wrote: > On Wed, Jun 20, 2018 at 12:07 PM Lukas Larsson wrote: > >> So if anyone is running a performance critical UDP server that has set >> the "recbuf" to a high value, I really recommend that you lower the >> "buffer" size to something close to the max expected packet size. >> > > That's interesting because the official documentation says: > > "It is recommended to have val(buffer) >= max(val(sndbuf),val(recbuf)) to > avoid performance issues because of unnecessary copying." > > > Maybe the documentation is wrong because that doesn't make much sense; > sndbuf and recbuf are sent to the kernel and are supposed to be bigger than > the user's buffer. > I think that part of the documentation is mainly written with TCP in mind, not UDP. Also, following the docs works fine if it weren't for the bug uncovered in this mail thread. I'll see what I can do about making the docs better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daron.ryan@REDACTED Thu Jun 21 12:08:13 2018 From: daron.ryan@REDACTED (Daron Ryan) Date: Thu, 21 Jun 2018 20:08:13 +1000 Subject: [erlang-questions] Erlang for Websites and Android Message-ID: Hi, I am just starting to learn Erlang. My plan is to develop a social media type application. I want it to have a dynamic web interface, which normally would be done with Javascript and later on an Android interface. I am not doing this for an organization. It is just something after hours. I have completed a part of the Erlang tutorial I have been browsing the web trying to find ideas on how to do my project. It seems there is some very good support for it. I even found a tool that compiles Erlang in Javascript but I am hoping to get some guidance what's effective and so on to save myself from too much experimentation. Are there any good examples of web applications for beginners to you can point me to? Good examples of Android apps? The simpler the better. Is there a system where you can write one set of client code in purely Erlang and then deploy it automatically to both Android and web interfaces? If I am on the wrong track by all means let me know. Thank you, Daron. -------------- next part -------------- An HTML attachment was scrubbed... URL: From seriy.pr@REDACTED Thu Jun 21 15:02:09 2018 From: seriy.pr@REDACTED (=?UTF-8?B?0KHQtdGA0LPQtdC5INCf0YDQvtGF0L7RgNC+0LI=?=) Date: Thu, 21 Jun 2018 15:02:09 +0200 Subject: [erlang-questions] UDP receive performance Message-ID: Regarding this suggestion in docs, I had some funny times because I blindly applied this suggestion on an application with large amount of not very active sockets. On my machine default settings for kernel buffers were [{sndbuf,87040},{recbuf,372480}] So, I ended up having 360kb userspace buffer per socket and I had like 10s of thousands of sockets. And that was very difficult to track down (`erlang:memory()` showed a lots of `binary` memory, while sum of sizes of binaries referenced by processes (via `process_info(P, binary)`) was 2 orders of magnitude smaller). I found the root cause only intuitively and I still don't know if there are any tool that is able to point me to a right direction. Even `[erlang:port_info(Port, memory) || Port <- erlang:ports()]` didn't show this memory. I also had prometheus BEAM allocators dashboard like this one https://github.com/deadtrickster/beam-dashboards/blob/master/BEAM-memory_allocators.png and it showed me 90% allocator utilization. So, nothing looked suspicious except just extremely high memory usage. > On Wed, Jun 20, 2018 at 12:07 PM Lukas Larsson wrote: > > > >> So if anyone is running a performance critical UDP server that has set > >> the "recbuf" to a high value, I really recommend that you lower the > >> "buffer" size to something close to the max expected packet size. > >> > > > > That's interesting because the official documentation says: > > > > "It is recommended to have val(buffer) >= max(val(sndbuf),val(recbuf)) to > > avoid performance issues because of unnecessary copying." > > > > > > Maybe the documentation is wrong because that doesn't make much sense; > > sndbuf and recbuf are sent to the kernel and are supposed to be bigger > than > > the user's buffer. > > > I think that part of the documentation is mainly written with TCP in mind, > not UDP. Also, following the docs works fine if it weren't for the bug > uncovered in this mail thread. I'll see what I can do about making the docs > better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Thu Jun 21 15:49:17 2018 From: z@REDACTED (Danil Zagoskin) Date: Thu, 21 Jun 2018 16:49:17 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: Hi! Sorry for the late reply. The patch works like a charm. Huge performance boost! Before: Thread alloc aux bifbusy_wait check_io emulator ets gc gc_full nif other port send sleep timers Stats per type: async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00% aux 0.04% 0.05% 0.00% 0.00% 2.02% 0.00% 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 0.00% 97.88% 0.00% dirty_cpu_sche 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00% dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00% poll 0.04% 0.00% 0.00% 0.00% 6.03% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 93.93% 0.00% scheduler 56.63% 0.80% 0.48% 16.13% 0.00% 4.07% 0.00% 1.24% 0.00% 0.00% 2.69% 14.89% 0.00% 2.80% 0.25% After: Thread alloc aux bifbusy_wait check_io emulator ets gc gc_full nif other port send sleep timers Stats per type: async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00% aux 0.00% 0.00% 0.00% 0.00% 14.57% 0.00% 0.00% 0.00% 0.00% 0.00% 1.24% 0.00% 0.00% 84.19% 0.00% dirty_cpu_sche 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00% dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00% poll 0.00% 0.00% 0.00% 0.00% 3.10% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 96.90% 0.00% scheduler 2.86% 0.78% 0.38% 57.22% 0.00% 2.81% 0.00% 0.49% 0.00% 0.00% 7.32% 9.87% 0.00% 18.08% 0.18% The machine is quad-core i5-6400@REDACTED with Linux 4.13.0 Load generated with this commands (10 senders, 10 readers on each address): [udptest:start_sender({239,9,9,X}, 3999) || X <- lists:seq(1,10)], [udptest:start_reader({239,9,9,X}, 3999) || X <- lists:seq(1,10), _ <- lists:seq(1, 10)]. Thank you for the awesome work! P.S. For some reason there is an increased check_io time in aux thread, but I think it's not critical and may be related to more often socket reads due to lots of idle time. On Wed, Jun 20, 2018 at 2:56 PM Jonas Falkevik < jonas.falkevik@REDACTED> wrote: > > On Mon, Jun 18, 2018 at 3:57 PM, Lukas Larsson wrote: > >> >> >>>> Something like >>> https://github.com/falkevik/otp/commit/444fb00ff2a9d1f40a8c66f48bea1cf3f07ca86c >>> ? >>> >> >> I was thinking something more like this: >> https://github.com/garazdawi/otp/tree/lukas/erts/realloc_thr_pref_shrink_th >> >> This makes it so that the absolute and relative single block shrink >> thesholds are respected for reallocs made on a remote scheduler. This >> should solve the problem that you have found also, through as I still >> haven't reproduced it I can't test that it actually solves it. >> >> > Are you adding the multicast network to the loop back interface? Using > some other interface that does not allow multicast traffic? > > I have been able to reproduce it in macOS Sierra (Darwin Kernel Version > 16.7.0: Fri Apr 27 17:59:46 PDT 2018; root:xnu-3789.73.13~1/RELEASE_X86_64 > x86_64) > and also on Ubuntu Linux 16.04 LTS with kernel 4.4.0-112-generic. > > Danil Zagoskin, do you have time to try and see if there is any change in > the behaviour for you with a patched system? > > > With the bugfix, the beam is spending most of its time in sched_spin_wait > on linux and sched_yield on macOS. > > Stats below are from OTP 21 and then OTP21 with bugfix on linux > > Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] > [async-threads:1] [hipe] > > Eshell V10.0 (abort with ^G) > 1> udptest:start_sender({239,9,9,9}, 3999). > <0.78.0> > 2> [udptest:start_reader({239,9,9,9}, 3999) || _ <- lists:seq(1, 40)]. > [<0.80.0>,<0.81.0>,<0.82.0>,<0.83.0>,<0.84.0>,<0.85.0>, > <0.86.0>,<0.87.0>,<0.88.0>,<0.89.0>,<0.90.0>,<0.91.0>, > <0.92.0>,<0.93.0>,<0.94.0>,<0.95.0>,<0.96.0>,<0.97.0>, > <0.98.0>,<0.99.0>,<0.100.0>,<0.101.0>,<0.102.0>,<0.103.0>, > <0.104.0>,<0.105.0>,<0.106.0>,<0.107.0>,<0.108.0>|...] > 3> msacc:start(10000), msacc:print(). > Average thread real-time : 10002138 us > Accumulated system run-time : 62576106 us > Average scheduler run-time : 7775309 us > > Thread aux check_io emulator gc other port > sleep > > Stats per thread: > async( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > aux( 1) 0.06% 0.96% 0.00% 0.00% 0.07% 0.00% > 98.90% > dirty_cpu_( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 9) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s(10) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > poll( 0) 0.00% 2.63% 0.00% 0.00% 0.00% 0.00% > 97.37% > scheduler( 1) 0.82% 0.00% 0.52% 0.35% 7.33% 67.91% > 23.08% > scheduler( 2) 0.76% 0.00% 0.79% 0.34% 7.30% 68.61% > 22.21% > scheduler( 3) 0.74% 0.00% 0.83% 0.33% 7.17% 68.54% > 22.38% > scheduler( 4) 0.76% 0.00% 0.72% 0.34% 7.38% 68.84% > 21.96% > scheduler( 5) 0.79% 0.00% 0.82% 0.34% 7.21% 68.94% > 21.90% > scheduler( 6) 0.80% 0.00% 0.74% 0.31% 7.15% 68.56% > 22.44% > scheduler( 7) 0.87% 0.00% 0.72% 0.37% 7.24% 68.43% > 22.36% > scheduler( 8) 0.77% 0.00% 0.68% 0.35% 7.41% 69.02% > 21.77% > > Stats per type: > async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > aux 0.06% 0.96% 0.00% 0.00% 0.07% 0.00% > 98.90% > dirty_cpu_sche 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > poll 0.00% 2.63% 0.00% 0.00% 0.00% 0.00% > 97.37% > scheduler 0.79% 0.00% 0.73% 0.34% 7.27% 68.61% > 22.26% > ok > 4> > > > $ ../src/udp_performace_bug_fix/bin/erl > Erlang/OTP 21 [RELEASE CANDIDATE 2] [erts-9.3.2] [source] [64-bit] > [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] > > Eshell V9.3.2 (abort with ^G) > 1> udptest:start_sender({239,9,9,9}, 3999). > <0.78.0> > 2> [udptest:start_reader({239,9,9,9}, 3999) || _ <- lists:seq(1, 40)]. > [<0.80.0>,<0.81.0>,<0.82.0>,<0.83.0>,<0.84.0>,<0.85.0>, > <0.86.0>,<0.87.0>,<0.88.0>,<0.89.0>,<0.90.0>,<0.91.0>, > <0.92.0>,<0.93.0>,<0.94.0>,<0.95.0>,<0.96.0>,<0.97.0>, > <0.98.0>,<0.99.0>,<0.100.0>,<0.101.0>,<0.102.0>,<0.103.0>, > <0.104.0>,<0.105.0>,<0.106.0>,<0.107.0>,<0.108.0>|...] > 3> msacc:start(10000), msacc:print(). > Average thread real-time : 10000416 us > Accumulated system run-time : 19541445 us > Average scheduler run-time : 2279822 us > > Thread aux check_io emulator gc other port > sleep > > Stats per thread: > async( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > aux( 1) 0.00% 10.72% 0.00% 0.00% 0.00% 0.00% > 89.28% > dirty_cpu_( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_cpu_( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s( 9) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_s(10) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > poll( 0) 0.00% 2.31% 0.00% 0.00% 0.00% 0.00% > 97.69% > scheduler( 1) 0.46% 0.00% 0.43% 0.14% 17.57% 4.07% > 77.33% > scheduler( 2) 0.50% 0.00% 0.44% 0.15% 17.13% 4.21% > 77.57% > scheduler( 3) 0.49% 0.00% 0.47% 0.16% 17.07% 4.35% > 77.46% > scheduler( 4) 0.45% 0.00% 0.42% 0.14% 16.25% 4.02% > 78.72% > scheduler( 5) 0.48% 0.00% 0.42% 0.14% 17.04% 4.04% > 77.88% > scheduler( 6) 0.46% 0.00% 0.39% 0.14% 16.57% 3.77% > 78.67% > scheduler( 7) 0.53% 0.00% 1.88% 0.27% 17.60% 8.22% > 71.50% > scheduler( 8) 0.47% 0.00% 0.42% 0.15% 16.60% 3.86% > 78.50% > > Stats per type: > async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > aux 0.00% 10.72% 0.00% 0.00% 0.00% 0.00% > 89.28% > dirty_cpu_sche 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% > 100.00% > poll 0.00% 2.31% 0.00% 0.00% 0.00% 0.00% > 97.69% > scheduler 0.48% 0.00% 0.61% 0.16% 16.98% 4.57% > 77.20% > ok > > Jonas > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Thu Jun 21 15:56:32 2018 From: z@REDACTED (Danil Zagoskin) Date: Thu, 21 Jun 2018 16:56:32 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: Hmm, seems like the value should be very close to expected size. Setting buffer to 2048 did not change anything significantly. The results in my previous message are with {buffer, 2048} option left from earlier experiments. On Wed, Jun 20, 2018 at 8:06 PM Lukas Larsson wrote: > On Wed, Jun 20, 2018 at 1:56 PM Jonas Falkevik < > jonas.falkevik@REDACTED> wrote: > >> >> Are you adding the multicast network to the loop back interface? Using >> some other interface that does not allow multicast traffic? >> >> > I though I had, but apparently not.... I managed to reproduce the test > case now. > > I also spent some time staring at the code in the inet_driver and realized > what the problem was in there. It would seem that a performance/feature fix > for SCTP in R15B inadvertently introduced this behaviour for UDP :( I've > pushed a bug fix to that problem in the inet_driver to the same branch. In > my tests the scheduler utilization goes form 71% to about 4% with both > patches. With the fix in the inet driver, my allocator patch does not > really do any difference for this specific testcase, but I'll keep that fix > anyways as it is a "good thing". I should probably also add a benchmark for > this so that it does not happen again.... > > I'd also like to add that changing the test-case to use a smaller > user-space buffer also has the same effect. So if anyone is running a > performance critical UDP server that has set the "recbuf" to a high value, > I really recommend that you lower the "buffer" size to something close to > the max expected packet size. In the example that Danil provided I applied > the following patch: > > diff --git a/udptest.erl b/udptest.erl > index 16a1798..4edeef0 100644 > --- a/udptest.erl > +++ b/udptest.erl > @@ -33,7 +33,9 @@ send_packet(ID, S, Addr, Port) -> > > start_reader(Addr, Port) -> > GwIP = {0,0,0,0}, % {127,0,0,1}, > - Common = > [binary,{reuseaddr,true},{recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], > + Common = [binary,{reuseaddr,true}, > + {buffer,1500}, %% 1500 is just an example value, don't just > copy this. You need to know what you max UDP packet size will be. > + {recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], > Multicast = > [{multicast_ttl,4},{multicast_loop,true},{ip,Addr},{add_membership,{Addr,GwIP}}], > Options = Common ++ Multicast, > spawn(fun() -> run_reader(Port, Options) end). > > and the scheduler utilization dropped to about 4% there as well. > > Lukas > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From rveyland2@REDACTED Thu Jun 21 16:31:44 2018 From: rveyland2@REDACTED (=?utf-8?Q?Ren=C3=A9_Veyland_Mobile?=) Date: Thu, 21 Jun 2018 16:31:44 +0200 Subject: [erlang-questions] [OTP 19.3] [Diameter 1.12.2] Handle_request with error: reply not taken into account Message-ID: <5b2bb6d1.1c69fb81.795bd.8063@mx.google.com> Dear Erlang Community, I?m a beginner in Erlang, and i have a question on diameter stack behavior. I ?m working on a sw acting as diameter peer server with a S6A app. I have noticed that when an diameter request is received with an error (for example an additional AVP not supported by dict), even if the packet provided in return value of the handle_request call back contains a valid answer with an AVP result-code=Success and header with errors=[] and is_error=false, the diameter stack return to client peer a diameter packet with an error 5001 (unsupported_avp) which is corresponding to the initial error found on request decoding (and not the answer sent in {reply, Reply}. If the initial diameter request is valid verus dict, then the answer provided by callback is sent back to client peer as expected. I would like to confirm that it is due to a bug from my sw (most probably?? ) So I have some questions?: - Is it a normal of diameter stack behavior o if yes how to avoid/bypass it?? (how to send a valid answer even if request is seen in error) o If not?: on which criteria a predefined answer is sent back to the transport layer instead of the application?s callback one?? - More globally: What is the simplest way to debug diameter stack in order to verify that data sent back by callback handler are ??as expected???? Thx a lot by advance for your support. BR Ren? Veyland Provenance?: Courrier pour Windows 10 -------------- next part -------------- An HTML attachment was scrubbed... URL: From eshikafe@REDACTED Thu Jun 21 17:22:38 2018 From: eshikafe@REDACTED (austin aigbe) Date: Thu, 21 Jun 2018 16:22:38 +0100 Subject: [erlang-questions] Erlang for Websites and Android In-Reply-To: References: Message-ID: Checkout Nitrogen: http://nitrogenproject.com/ or Zotonic: http://zotonic.com/ Regards, Austin On Thu, Jun 21, 2018 at 11:36 AM Daron Ryan wrote: > Hi, > > I am just starting to learn Erlang. My plan is to develop a social media > type application. I want it to have a dynamic web interface, which normally > would be done with Javascript and later on an Android interface. I am not > doing this for an organization. It is just something after hours. I have > completed a part of the Erlang tutorial I have been browsing the web > trying to find ideas on how to do my project. It seems there is some very > good support for it. I even found a tool that compiles Erlang in Javascript > but I am hoping to get some guidance what's effective and so on to save > myself from too much experimentation. > > Are there any good examples of web applications for beginners to you can > point me to? > Good examples of Android apps? The simpler the better. > > Is there a system where you can write one set of client code in purely > Erlang and then deploy it automatically to both Android and web interfaces? > > If I am on the wrong track by all means let me know. > > Thank you, > Daron. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandrashekhar.mullaparthi@REDACTED Thu Jun 21 18:31:42 2018 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Thu, 21 Jun 2018 17:31:42 +0100 Subject: [erlang-questions] [OTP 19.3] [Diameter 1.12.2] Handle_request with error: reply not taken into account In-Reply-To: <5b2bb6d1.1c69fb81.795bd.8063@mx.google.com> References: <5b2bb6d1.1c69fb81.795bd.8063@mx.google.com> Message-ID: Hi Ren?, On 21 June 2018 at 15:31, Ren? Veyland Mobile wrote: > Dear Erlang Community, > > > > I?m a beginner in Erlang, and i have a question on diameter stack behavior. > > > > I ?m working on a sw acting as diameter peer server with a S6A app. > > > > I have noticed that when an diameter request is received with an error > (for example an additional AVP not supported by dict), even if the packet > provided in return value of the handle_request call back contains a valid > answer with an AVP result-code=Success and header with errors=[] and > is_error=false, the diameter stack return to client peer a diameter packet > with an error 5001 (unsupported_avp) which is corresponding to the initial > error found on request decoding (and not the answer sent in {reply, Reply}. > > > > If the initial diameter request is valid verus dict, then the answer > provided by callback is sent back to client peer as expected. > > > > I would like to confirm that it is due to a bug from my sw (most probably > ? ) > > > > So I have some questions : > > - Is it a normal of diameter stack behavior > - if yes how to avoid/bypass it ? (how to send a valid answer even > if request is seen in error) > > This code works for me in the handle_request/3 callback: handle_request(#diameter_packet{msg = Req, transport_data = TD}, _SvcName, {_, Caps}) when is_record(Req, 'diameter_gx_CCR') -> #diameter_caps{origin_host = {OH,_}, origin_realm = {OR,_}} = Caps, #'diameter_gx_CCR'{'Session-Id' = Id, 'Auth-Application-Id' = AuthAppId, 'CC-Request-Type' = ReqType, 'CC-Request-Number' = ReqNum} = Req, PolicyMap = get_policy(Req), CCA = #'diameter_gx_CCA'{'Result-Code' = 2001, 'Auth-Application-Id' = AuthAppId, 'Origin-Host' = OH, 'Origin-Realm' = OR, 'Session-Id' = Id, 'CC-Request-Type' = ReqType, 'CC-Request-Number' = ReqNum, 'Supported-Features' = get_supported_features(), 'QoS-Information' = get_qos(PolicyMap), 'Charging-Rule-Install' = get_crbn(PolicyMap) }, ReplyPacket = #diameter_packet{msg = CCA, header = #diameter_header{}, errors = false, transport_data = TD}, {reply, ReplyPacket}; Also, when initialising the service, I tend to use the {strict_mbit, false} option this is a frequent annoyance. > - More globally: What is the simplest way to debug diameter stack in > order to verify that data sent back by callback handler are ? as > expected ? ? > > I've also found debugging diameter applications to be quite hard. I've had to resort to using dbg to trace calls in the diameter_codec module to figure out what was going wrong. regards, Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From unexplained@REDACTED Fri Jun 22 03:07:47 2018 From: unexplained@REDACTED (Michael Stellar) Date: Fri, 22 Jun 2018 08:07:47 +0700 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: Looks good, does these patch already being committed upstream? On Thu, Jun 21, 2018 at 8:56 PM Danil Zagoskin wrote: > Hmm, seems like the value should be very close to expected size. > Setting buffer to 2048 did not change anything significantly. > The results in my previous message are with {buffer, 2048} option left > from earlier experiments. > > On Wed, Jun 20, 2018 at 8:06 PM Lukas Larsson wrote: > >> On Wed, Jun 20, 2018 at 1:56 PM Jonas Falkevik < >> jonas.falkevik@REDACTED> wrote: >> >>> >>> Are you adding the multicast network to the loop back interface? Using >>> some other interface that does not allow multicast traffic? >>> >>> >> I though I had, but apparently not.... I managed to reproduce the test >> case now. >> >> I also spent some time staring at the code in the inet_driver and >> realized what the problem was in there. It would seem that a >> performance/feature fix for SCTP in R15B inadvertently introduced this >> behaviour for UDP :( I've pushed a bug fix to that problem in the >> inet_driver to the same branch. In my tests the scheduler utilization goes >> form 71% to about 4% with both patches. With the fix in the inet driver, my >> allocator patch does not really do any difference for this specific >> testcase, but I'll keep that fix anyways as it is a "good thing". I should >> probably also add a benchmark for this so that it does not happen again.... >> >> I'd also like to add that changing the test-case to use a smaller >> user-space buffer also has the same effect. So if anyone is running a >> performance critical UDP server that has set the "recbuf" to a high value, >> I really recommend that you lower the "buffer" size to something close to >> the max expected packet size. In the example that Danil provided I applied >> the following patch: >> >> diff --git a/udptest.erl b/udptest.erl >> index 16a1798..4edeef0 100644 >> --- a/udptest.erl >> +++ b/udptest.erl >> @@ -33,7 +33,9 @@ send_packet(ID, S, Addr, Port) -> >> >> start_reader(Addr, Port) -> >> GwIP = {0,0,0,0}, % {127,0,0,1}, >> - Common = >> [binary,{reuseaddr,true},{recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], >> + Common = [binary,{reuseaddr,true}, >> + {buffer,1500}, %% 1500 is just an example value, don't just >> copy this. You need to know what you max UDP packet size will be. >> + {recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], >> Multicast = >> [{multicast_ttl,4},{multicast_loop,true},{ip,Addr},{add_membership,{Addr,GwIP}}], >> Options = Common ++ Multicast, >> spawn(fun() -> run_reader(Port, Options) end). >> >> and the scheduler utilization dropped to about 4% there as well. >> >> Lukas >> >> > > -- > Danil Zagoskin | z@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Fri Jun 22 09:24:05 2018 From: pablo.platt@REDACTED (pablo platt) Date: Fri, 22 Jun 2018 10:24:05 +0300 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: Does this bug only affect multicast UDP or also normal UDP? What values should I use for buffer and recbuf in a UDP socket receiving 1Mbps with 1500 Bytes MTU? I thought I need N*MTU recbuf so the pid will be able to handle a small burst of UDP packets. This is what I currently have: gen_udp:open(0, [binary, {active, once}, {recbuf, 16*1024}]) On Fri, Jun 22, 2018 at 4:07 AM, Michael Stellar wrote: > Looks good, does these patch already being committed upstream? > > On Thu, Jun 21, 2018 at 8:56 PM Danil Zagoskin wrote: > >> Hmm, seems like the value should be very close to expected size. >> Setting buffer to 2048 did not change anything significantly. >> The results in my previous message are with {buffer, 2048} option left >> from earlier experiments. >> >> On Wed, Jun 20, 2018 at 8:06 PM Lukas Larsson wrote: >> >>> On Wed, Jun 20, 2018 at 1:56 PM Jonas Falkevik < >>> jonas.falkevik@REDACTED> wrote: >>> >>>> >>>> Are you adding the multicast network to the loop back interface? Using >>>> some other interface that does not allow multicast traffic? >>>> >>>> >>> I though I had, but apparently not.... I managed to reproduce the test >>> case now. >>> >>> I also spent some time staring at the code in the inet_driver and >>> realized what the problem was in there. It would seem that a >>> performance/feature fix for SCTP in R15B inadvertently introduced this >>> behaviour for UDP :( I've pushed a bug fix to that problem in the >>> inet_driver to the same branch. In my tests the scheduler utilization goes >>> form 71% to about 4% with both patches. With the fix in the inet driver, my >>> allocator patch does not really do any difference for this specific >>> testcase, but I'll keep that fix anyways as it is a "good thing". I should >>> probably also add a benchmark for this so that it does not happen again.... >>> >>> I'd also like to add that changing the test-case to use a smaller >>> user-space buffer also has the same effect. So if anyone is running a >>> performance critical UDP server that has set the "recbuf" to a high value, >>> I really recommend that you lower the "buffer" size to something close to >>> the max expected packet size. In the example that Danil provided I applied >>> the following patch: >>> >>> diff --git a/udptest.erl b/udptest.erl >>> index 16a1798..4edeef0 100644 >>> --- a/udptest.erl >>> +++ b/udptest.erl >>> @@ -33,7 +33,9 @@ send_packet(ID, S, Addr, Port) -> >>> >>> start_reader(Addr, Port) -> >>> GwIP = {0,0,0,0}, % {127,0,0,1}, >>> - Common = [binary,{reuseaddr,true},{recbuf,2*1024*1024},inet,{ >>> read_packets,100},{active,500}], >>> + Common = [binary,{reuseaddr,true}, >>> + {buffer,1500}, %% 1500 is just an example value, don't just >>> copy this. You need to know what you max UDP packet size will be. >>> + {recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], >>> Multicast = [{multicast_ttl,4},{multicast_loop,true},{ip,Addr},{add_ >>> membership,{Addr,GwIP}}], >>> Options = Common ++ Multicast, >>> spawn(fun() -> run_reader(Port, Options) end). >>> >>> and the scheduler utilization dropped to about 4% there as well. >>> >>> Lukas >>> >>> >> >> -- >> Danil Zagoskin | z@REDACTED >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Jun 22 10:14:15 2018 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 22 Jun 2018 10:14:15 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On Fri, 22 Jun 2018, 03:08 Michael Stellar, wrote: > Looks good, does these patch already being committed upstream? > No, and it will be a couple of weeks until it is. I'll post here when it is merged. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Jun 22 10:27:02 2018 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 22 Jun 2018 10:27:02 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On Fri, 22 Jun 2018, 09:24 pablo platt, wrote: > Does this bug only affect multicast UDP or also normal UDP? > It effects reception of all UDP messages. > What values should I use for buffer and recbuf in a UDP socket receiving > 1Mbps with 1500 Bytes MTU? > I thought I need N*MTU recbuf so the pid will be able to handle a small > burst of UDP packets. > There is no need to set buffer larger than your MTU. The recbuf however should be large enough to handle any bursts that may happen. This is what I currently have: > gen_udp:open(0, [binary, {active, once}, {recbuf, 16*1024}]) > I doubt that you will see any large performance differences by setting the buffer size to, let's say 2*1024. But it will as always depend on your application. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rveyland2@REDACTED Fri Jun 22 09:27:18 2018 From: rveyland2@REDACTED (=?utf-8?Q?Ren=C3=A9_Veyland_Mobile?=) Date: Fri, 22 Jun 2018 09:27:18 +0200 Subject: [erlang-questions] [OTP 19.3] [Diameter 1.12.2] Handle_requestwith error: reply not taken into account In-Reply-To: References: <5b2bb6d1.1c69fb81.795bd.8063@mx.google.com> Message-ID: <5b2ca4d6.1c69fb81.42e25.f323@mx.google.com> Hi Chandru Thx for answer. I will crosscheck my sw again even if very similar. The issue is only visible if recevied diameter packet as an error raised. For debuging, i will try your method too. Br Ren? Provenance?: Courrier pour Windows 10 De?: Chandru Envoy? le?:jeudi 21 juin 2018 18:32 ??: Ren? Veyland Mobile Cc?: erlang-questions@REDACTED Objet?:Re: [erlang-questions] [OTP 19.3] [Diameter 1.12.2] Handle_requestwith error: reply not taken into account Hi Ren?, On 21 June 2018 at 15:31, Ren? Veyland Mobile wrote: Dear Erlang Community, ? I?m a beginner in Erlang, and i have a question on diameter stack behavior. ? I ?m working on a sw acting as diameter peer server with a S6A app. ? I have noticed that when an diameter request is received with an error (for example an additional AVP not supported by dict), even if the packet provided in return value of the handle_request call back contains a valid answer with an AVP result-code=Success and header with errors=[] and is_error=false, the diameter stack return to client peer a diameter packet with an error 5001 (unsupported_avp) which is corresponding to the initial error found on request decoding (and not the answer sent in {reply, Reply}. ? If the initial diameter request is valid verus dict, then the answer provided by callback is sent back to client peer as expected. ? I would like to confirm that it is due to a bug from my sw (most probably?? ) ? So I have some questions?: ? Is it a normal of diameter stack behavior o ?if yes how to avoid/bypass it?? (how to send a valid answer even if request is seen in error) This code works for me in the handle_request/3 callback: handle_request(#diameter_packet{msg = Req, transport_data = TD}, _SvcName, {_, Caps}) ? when is_record(Req, 'diameter_gx_CCR') -> ? ? #diameter_caps{origin_host = {OH,_}, ? ? ? ? ? ? ? ? ? ?origin_realm = {OR,_}} ? ? ? ? ? ? ? ?= Caps, ? ? #'diameter_gx_CCR'{'Session-Id' ? ? ? ? ?= Id, ? ? ? ? ? ? ? ? ? ? ? ?'Auth-Application-Id' = AuthAppId, ? ? ? ? ? ? ? ? ? ? ? ?'CC-Request-Type' ? ? = ReqType, ? ? ? ? ? ? ? ? ? ? ? ?'CC-Request-Number' ? = ReqNum} ?= Req, ? ? PolicyMap = get_policy(Req), ? ? CCA ? = #'diameter_gx_CCA'{'Result-Code' ? ? ? ? ? ? ? ? ? ? ?= 2001, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'Auth-Application-Id' ? ? ? ? ? ? ?= AuthAppId, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'Origin-Host' ? ? ? ? ? ? ? ? ? ? ?= OH, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'Origin-Realm' ? ? ? ? ? ? ? ? ? ? = OR, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'Session-Id' ? ? ? ? ? ? ? ? ? ? ? = Id, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'CC-Request-Type' ? ? ? ? ? ? ? ? ?= ReqType, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'CC-Request-Number' ? ? ? ? ? ? ? ?= ReqNum, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'Supported-Features' ? ? ? ? ? ? ? = get_supported_features(), ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'QoS-Information' ? ? ? ? ? ? ? ? ?= get_qos(PolicyMap), ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?'Charging-Rule-Install' ? ? ? ? ? ?= get_crbn(PolicyMap) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? }, ? ? ReplyPacket = #diameter_packet{msg = CCA, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?header = #diameter_header{}, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?errors = false, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?transport_data = TD}, ? ? {reply, ReplyPacket}; Also, when initialising the service, I tend to use the {strict_mbit, false} option this is a frequent annoyance. o More globally: What is the simplest way to debug diameter stack in order to verify that data sent back by callback handler are ??as expected???? I've also found debugging diameter applications to be quite hard. I've had to resort to using dbg to trace calls in the diameter_codec module to figure out what was going wrong. regards, Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andras.Bekes@REDACTED Fri Jun 22 13:53:37 2018 From: Andras.Bekes@REDACTED (Bekes, Andras G) Date: Fri, 22 Jun 2018 11:53:37 +0000 Subject: [erlang-questions] inet:getstat on non-socket crashes the VM? Message-ID: Hi Erlangers, I noticed that calling inet:getstat/1 is a dangerous operation, as it can kill the whole VM if used inappropriately. It is easy to reproduce: Eshell V9.3 (abort with ^G) 1> application:start(sasl). ok 2> [inet:getstat(X) || X<-erlang:ports()]. Failed to write to erl_child_setup: 9 Crash dump is being written to: erl_crash.dump...done Without sasl, it doesn't kill the VM, but the terminal gets into a faulty state and further work is not possible. I know calling it on a non-socket is an erroneous action, but crashing the VM doesn't seem to be the most appropriate consequence. What do you think? Regards, Andras G. Bekes ________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Fri Jun 22 17:00:32 2018 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 22 Jun 2018 16:00:32 +0100 Subject: [erlang-questions] Sharing a UDP socket between processes Message-ID: I need to send fire-and-forget UDP messages to a particular destination. I don't plan on receiving any incoming messages. I need to be able to send messages from several thousand Erlang processes. Ordinarily, I'd wrap the socket in a gen_server, but is there really any need? Can I simply stash the socket in an ETS table and use it directly? What possible downsides are there to this? Or would it be sensible to simply open/send/close a UDP socket each time? Each process only needs to send a message at startup, and then every 2 minutes or so. But: there will be several thousand of these processes spawned at once. From dmkolesnikov@REDACTED Fri Jun 22 17:19:47 2018 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 22 Jun 2018 18:19:47 +0300 Subject: [erlang-questions] Sharing a UDP socket between processes In-Reply-To: References: Message-ID: <65016F52-DE3E-4F0E-A904-7710351BE80F@gmail.com> Hello, > On 22 Jun 2018, at 18.00, Roger Lipscombe wrote: > > I need to send fire-and-forget UDP messages to a particular > destination. I don't plan on receiving any incoming messages. > > I need to be able to send messages from several thousand Erlang > processes. Ordinarily, I'd wrap the socket in a gen_server, but is > there really any need? You should design system where resource ownership is explicitly managed. gen_server is simply allows to build a fault tolerance solution. > > Can I simply stash the socket in an ETS table and use it directly? > What possible downsides are there to this? A single gen_server process is able to process over 80K messages per second before it became a bottleneck. I would start with single process and start scanning on the needs. Personally, I do not like usage of ETS as shared resources pool. I prefer to go with Process pool(s) manages by supervisor(s). One process owns a resource (UDP socket) other Child processes uses this resource. My reasoning simple here, supervisors helps you to react on failures. > > Or would it be sensible to simply open/send/close a UDP socket each > time? Each process only needs to send a message at startup, and then > every 2 minutes or so. But: there will be several thousand of these > processes spawned at once. I would not recommend this option because single open / close causes resource allocation from kernel. However, some monitoring solution uses this principle. Best Regards, Dmitry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From t@REDACTED Fri Jun 22 22:48:06 2018 From: t@REDACTED (Tristan Sloughter) Date: Fri, 22 Jun 2018 14:48:06 -0600 Subject: [erlang-questions] with Erlang R21 rebar3 build fails In-Reply-To: <1529566907.19459.33.camel@ericsson.com> References: <39CFC171-696D-4FCE-B760-3BF1F4AF8F57@gmail.com> <20180620232157.GG8817@ferdmbp.local> <1529566907.19459.33.camel@ericsson.com> Message-ID: <1529700486.2643491.1417380040.58A1BC32@webmail.messagingengine.com> And this allowed it to then set the mode which would actually succeed even though setting the owner did not. relx relied on simply copying the file_info from the original file to the copy it makes when including erts in a release. Without this files like `bin/erl` that need to be executable wouldn't be. My guess is that before 21 it would work for setting the mode and timestamp but not the owner but continue along like everything worked. While today since the owner is set first in `prim_file:write_file_info` and fails it never sets the mode, meaning we also can't simply ignore this failure like was happening before 21. We will patch rebar3/relx to catch this and attempt to update only the mode for a quick fix, or simply modify `ec_file:copy` in erlware_commons to support an options list of what file info to copy instead of attempting to copy it all requiring rebar3/relx to handle the failure. Tristan On Thu, Jun 21, 2018, at 1:41 AM, John H?gberg wrote: > Hi, > > This is caused by fixing a bug in OTP 21; file:write_file_info and > friends used to report success on owner/group changes regardless of > whether it worked or not. > > We marked this as an incompatibility in the release notes under OTP- > 15118. > > Regards, > John H?gberg > > On ons, 2018-06-20 at 19:21 -0400, Fred Hebert wrote: > > On 06/20, Mark Geib wrote: > > > > > > and rebar3.crashdump is: > > > > > > geibs-MacBook-Pro:json2splunk geib$ cat rebar3.crashdump > > > Error: {badmatch,{error,eperm}} > > > [{rlx_prv_overlay,write_template,3, > > > ?????????????????[{file,"/home/tristan/Devel/rebar3/_build/default/ > > > lib/relx/src/rlx_prv_overlay.erl"}, > > > ??????????????????{line,498}]}, > > > ... > > > > > > Now switching back to erlang R20 I get a successful build > > > > > > > > > I assume since rebar3 is the same exec, that this is a potential > > > issue with R21. > > > Any ideas.??? > > > > > See http://erlang.org/pipermail/erlang-questions/2018-June/095614.htm > > l > > > > OTP-21.0 sees the entire file drivers rewritten. The bug you have > > seems? > > to have been reported to the OTP team during OTP-21-rc2. > > > > I don't know exactly what happened between OTP-21-rc2 and OTP-21.0, > > and? > > if you'd need to run with a rebar3 built specifically on OTP-21.0 > > for? > > things to work fine. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From t@REDACTED Sat Jun 23 01:02:28 2018 From: t@REDACTED (Tristan Sloughter) Date: Fri, 22 Jun 2018 17:02:28 -0600 Subject: [erlang-questions] with Erlang R21 rebar3 build fails In-Reply-To: <1529700486.2643491.1417380040.58A1BC32@webmail.messagingengine.com> References: <39CFC171-696D-4FCE-B760-3BF1F4AF8F57@gmail.com> <20180620232157.GG8817@ferdmbp.local> <1529566907.19459.33.camel@ericsson.com> <1529700486.2643491.1417380040.58A1BC32@webmail.messagingengine.com> Message-ID: <1529708548.2684459.1417484216.50D1CB16@webmail.messagingengine.com> See https://github.com/erlang/rebar3/pull/1825 for a branch that may work and I'd like to get verification from those who are seeing this issue that it fixes the issue for them. Please check not just that you can build the release but that it runs successfully and can be tarred. -- Tristan Sloughter "I am not a crackpot" - Abe Simpson t@REDACTED On Fri, Jun 22, 2018, at 2:48 PM, Tristan Sloughter wrote: > And this allowed it to then set the mode which would actually succeed > even though setting the owner did not. > > relx relied on simply copying the file_info from the original file to > the copy it makes when including erts in a release. Without this files > like `bin/erl` that need to be executable wouldn't be. > > My guess is that before 21 it would work for setting the mode and > timestamp but not the owner but continue along like everything worked. > While today since the owner is set first in `prim_file:write_file_info` > and fails it never sets the mode, meaning we also can't simply ignore > this failure like was happening before 21. > > We will patch rebar3/relx to catch this and attempt to update only the > mode for a quick fix, or simply modify `ec_file:copy` in erlware_commons > to support an options list of what file info to copy instead of > attempting to copy it all requiring rebar3/relx to handle the failure. > > Tristan > > On Thu, Jun 21, 2018, at 1:41 AM, John H?gberg wrote: > > Hi, > > > > This is caused by fixing a bug in OTP 21; file:write_file_info and > > friends used to report success on owner/group changes regardless of > > whether it worked or not. > > > > We marked this as an incompatibility in the release notes under OTP- > > 15118. > > > > Regards, > > John H?gberg > > > > On ons, 2018-06-20 at 19:21 -0400, Fred Hebert wrote: > > > On 06/20, Mark Geib wrote: > > > > > > > > and rebar3.crashdump is: > > > > > > > > geibs-MacBook-Pro:json2splunk geib$ cat rebar3.crashdump > > > > Error: {badmatch,{error,eperm}} > > > > [{rlx_prv_overlay,write_template,3, > > > > ?????????????????[{file,"/home/tristan/Devel/rebar3/_build/default/ > > > > lib/relx/src/rlx_prv_overlay.erl"}, > > > > ??????????????????{line,498}]}, > > > > ... > > > > > > > > Now switching back to erlang R20 I get a successful build > > > > > > > > > > > > I assume since rebar3 is the same exec, that this is a potential > > > > issue with R21. > > > > Any ideas.??? > > > > > > > See http://erlang.org/pipermail/erlang-questions/2018-June/095614.htm > > > l > > > > > > OTP-21.0 sees the entire file drivers rewritten. The bug you have > > > seems? > > > to have been reported to the OTP team during OTP-21-rc2. > > > > > > I don't know exactly what happened between OTP-21-rc2 and OTP-21.0, > > > and? > > > if you'd need to run with a rebar3 built specifically on OTP-21.0 > > > for? > > > things to work fine. > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From soverdor@REDACTED Fri Jun 22 23:41:09 2018 From: soverdor@REDACTED (Sam Overdorf) Date: Fri, 22 Jun 2018 14:41:09 -0700 Subject: [erlang-questions] String type Message-ID: Has anyone considered making string a type and not a list of chars. I seem to have a lot of trouble when a list is a bunch of string objects and I start taking it apart with [H|T] = List.. When processing the last string in the list I end up taking apart the individual characters of the string. If I do a type-check it tells me it is a list. I usually have to do a work around to handle this. If it was a type I would easily know when I am done with the list. Thanks, Sam soverdor@REDACTED From Oliver.Korpilla@REDACTED Sat Jun 23 09:07:23 2018 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Sat, 23 Jun 2018 09:07:23 +0200 Subject: [erlang-questions] gen_statem: Join states? Message-ID: Hello. In the system we're building we're using gen_statem processes to keep track of signalling scenarios, essentially series of different types of messages from various sources. For linear or branching scenarios gen_statem is well prepared. But there's one thing that seemed to be missing and that is support for joins? I recently reviewed a developer's code that implemented receiving two different messages that could arrive in any order as four state functions - receiving A first (1), then receiving B (2), then entering an end state; or receiving B first (3), then receiving A (4), then entering an end state. The code entering the end state was duplicated in both (2) and (4). What essentially was needed was a "check list" of which messages had already arrived to determine the transition into the end state. Such "check list" scenarios are quite common in our work (and can grow to be more complex) and I couldn't find a direct support for it (though I was limiting myself to state_function mode), so we're now starting a separate "join process" - a gen_server that keeps and maintain the join list - and do the decision now from a single wait state. Given how light-weight Erlang processes are this works fine. What prevented me from doing this through data maintained between calls was that I would have to partition the state into information for tracking the join and the actual information kept to process the signals. This seemed less generic and clean. I'm also aware that had I rewritten the whole state machine to handle_event functions and utilized a complex state this could have helped achieve the same. What is essentially is needed, from my POV, is a way to track information pertaining to state transitions separate from information for processing incoming events, and complex states seem to do that...? What is your preferred way of doing this? Is it possible to somehow support this at the library level? Are handle_event/complex states the way to go in such situations? Thank you and regards, Oliver From fchschneider@REDACTED Sat Jun 23 10:19:30 2018 From: fchschneider@REDACTED (Frans Schneider) Date: Sat, 23 Jun 2018 10:19:30 +0200 Subject: [erlang-questions] gen_statem: Join states? In-Reply-To: References: Message-ID: <59144101-38c7-dfd5-ab92-a8256a161e60@gmail.com> I use gen_statem's complex state often for exact the situation you describe. For example, I use this state: -record(state, {reliability_level :: best_effort | reliable, state1 :: idle | announcing, state2 :: waiting | must_repair}). Now with handle_event/4, it is very easy to handle any of these sub-states, even with enter calls. To me, this is what makes gen_statem so nice. Do mind that one has to make use of generic time-outs and not state time-outs, with complex states. Frans On 06/23/2018 09:07 AM, Oliver Korpilla wrote: > Hello. > > In the system we're building we're using gen_statem processes to keep track of signalling scenarios, essentially series of different types of messages from various sources. For linear or branching scenarios gen_statem is well prepared. But there's one thing that seemed to be missing and that is support for joins? > > I recently reviewed a developer's code that implemented receiving two different messages that could arrive in any order as four state functions - receiving A first (1), then receiving B (2), then entering an end state; or receiving B first (3), then receiving A (4), then entering an end state. The code entering the end state was duplicated in both (2) and (4). What essentially was needed was a "check list" of which messages had already arrived to determine the transition into the end state. > > Such "check list" scenarios are quite common in our work (and can grow to be more complex) and I couldn't find a direct support for it (though I was limiting myself to state_function mode), so we're now starting a separate "join process" - a gen_server that keeps and maintain the join list - and do the decision now from a single wait state. Given how light-weight Erlang processes are this works fine. > > What prevented me from doing this through data maintained between calls was that I would have to partition the state into information for tracking the join and the actual information kept to process the signals. This seemed less generic and clean. > > I'm also aware that had I rewritten the whole state machine to handle_event functions and utilized a complex state this could have helped achieve the same. What is essentially is needed, from my POV, is a way to track information pertaining to state transitions separate from information for processing incoming events, and complex states seem to do that...? > > What is your preferred way of doing this? Is it possible to somehow support this at the library level? Are handle_event/complex states the way to go in such situations? > > Thank you and regards, > Oliver > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From jesper.louis.andersen@REDACTED Sat Jun 23 12:21:36 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 23 Jun 2018 12:21:36 +0200 Subject: [erlang-questions] gen_statem: Join states? In-Reply-To: References: Message-ID: I usually stuff the events into a map and then have a handle_continue/2 like behavior to check if everything in the map is present. Simplified: event(a, State) -> check(State#{ a => true }); event(b, State) -> check(State#{ b => true }); ... check(#{ a := A, b := B} = State) -> end(A, B, State); check(#{} = State) -> {noreply, ...}. Usually, rather than storing true, I'd store the event data there, and rather than having a map() at the top-level I'd probably have a record. But the gist of the idea is there. A more advanced variant of this will have a defined state machine and valid transitions between them. Then check is also an invariant on the transitions validity and it becomes a (runtime) contract to check against. Aside: Idris can do this check at compile time, but since we don't have dependent types... On Sat, Jun 23, 2018 at 9:07 AM Oliver Korpilla wrote: > Hello. > > In the system we're building we're using gen_statem processes to keep > track of signalling scenarios, essentially series of different types of > messages from various sources. For linear or branching scenarios gen_statem > is well prepared. But there's one thing that seemed to be missing and that > is support for joins? > > I recently reviewed a developer's code that implemented receiving two > different messages that could arrive in any order as four state functions - > receiving A first (1), then receiving B (2), then entering an end state; or > receiving B first (3), then receiving A (4), then entering an end state. > The code entering the end state was duplicated in both (2) and (4). What > essentially was needed was a "check list" of which messages had already > arrived to determine the transition into the end state. > > Such "check list" scenarios are quite common in our work (and can grow to > be more complex) and I couldn't find a direct support for it (though I was > limiting myself to state_function mode), so we're now starting a separate > "join process" - a gen_server that keeps and maintain the join list - and > do the decision now from a single wait state. Given how light-weight Erlang > processes are this works fine. > > What prevented me from doing this through data maintained between calls > was that I would have to partition the state into information for tracking > the join and the actual information kept to process the signals. This > seemed less generic and clean. > > I'm also aware that had I rewritten the whole state machine to > handle_event functions and utilized a complex state this could have helped > achieve the same. What is essentially is needed, from my POV, is a way to > track information pertaining to state transitions separate from information > for processing incoming events, and complex states seem to do that...? > > What is your preferred way of doing this? Is it possible to somehow > support this at the library level? Are handle_event/complex states the way > to go in such situations? > > Thank you and regards, > Oliver > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Sat Jun 23 12:22:41 2018 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Sat, 23 Jun 2018 12:22:41 +0200 Subject: [erlang-questions] gen_statem: Join states? In-Reply-To: <59144101-38c7-dfd5-ab92-a8256a161e60@gmail.com> References: <59144101-38c7-dfd5-ab92-a8256a161e60@gmail.com> Message-ID: Hello, Frans. So your recommendation is to prefer handle_event state machines whenever it is foreseeable that a join state is required? I'm sorry, I didn't understand the intent behind your example. Thank you, Oliver ? ? Gesendet:?Samstag, 23. Juni 2018 um 10:19 Uhr Von:?"Frans Schneider" An:?"Oliver Korpilla" , erlang-questions@REDACTED Betreff:?Re: [erlang-questions] gen_statem: Join states? I use gen_statem's complex state often for exact the situation you describe. For example, I use this state: -record(state, {reliability_level :: best_effort | reliable, state1 :: idle | announcing, state2 :: waiting | must_repair}). Now with handle_event/4, it is very easy to handle any of these sub-states, even with enter calls. To me, this is what makes gen_statem so nice. Do mind that one has to make use of generic time-outs and not state time-outs, with complex states. Frans On 06/23/2018 09:07 AM, Oliver Korpilla wrote: > Hello. > > In the system we're building we're using gen_statem processes to keep track of signalling scenarios, essentially series of different types of messages from various sources. For linear or branching scenarios gen_statem is well prepared. But there's one thing that seemed to be missing and that is support for joins? > > I recently reviewed a developer's code that implemented receiving two different messages that could arrive in any order as four state functions - receiving A first (1), then receiving B (2), then entering an end state; or receiving B first (3), then receiving A (4), then entering an end state. The code entering the end state was duplicated in both (2) and (4). What essentially was needed was a "check list" of which messages had already arrived to determine the transition into the end state. > > Such "check list" scenarios are quite common in our work (and can grow to be more complex) and I couldn't find a direct support for it (though I was limiting myself to state_function mode), so we're now starting a separate "join process" - a gen_server that keeps and maintain the join list - and do the decision now from a single wait state. Given how light-weight Erlang processes are this works fine. > > What prevented me from doing this through data maintained between calls was that I would have to partition the state into information for tracking the join and the actual information kept to process the signals. This seemed less generic and clean. > > I'm also aware that had I rewritten the whole state machine to handle_event functions and utilized a complex state this could have helped achieve the same. What is essentially is needed, from my POV, is a way to track information pertaining to state transitions separate from information for processing incoming events, and complex states seem to do that...? > > What is your preferred way of doing this? Is it possible to somehow support this at the library level? Are handle_event/complex states the way to go in such situations? > > Thank you and regards, > Oliver > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From mononcqc@REDACTED Sat Jun 23 14:05:15 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Sat, 23 Jun 2018 08:05:15 -0400 Subject: [erlang-questions] String type In-Reply-To: References: Message-ID: <20180623120514.GJ8817@ferdmbp.local> On 06/22, Sam Overdorf wrote: >Has anyone considered making string a type and not a list of chars. > Yes, please see for example this discussion from earlier this month: http://erlang.org/pipermail/erlang-questions/2018-June/095572.html >I seem to have a lot of trouble when a list is a bunch of string >objects and I start taking it apart with [H|T] = List.. > > When processing the last string in the list I end up taking apart the >individual characters of the string. If I do a type-check it tells me >it is a list. > use string:to_graphemes(S) and you'll have a list of unicode-aware characters that can be iterated over, in an encoding-neutral format: 1> string:to_graphemes("??e?"). [223,8593,[101,778]] 2> string:to_graphemes(<<"??e?"/utf8>>). [223,8593,[101,778]] Regards, Fred. From kostis@REDACTED Sat Jun 23 14:07:02 2018 From: kostis@REDACTED (Kostis Sagonas) Date: Sat, 23 Jun 2018 14:07:02 +0200 Subject: [erlang-questions] String type In-Reply-To: References: Message-ID: On 06/22/2018 11:41 PM, Sam Overdorf wrote: > Has anyone considered making string a type and not a list of chars. > > I seem to have a lot of trouble when a list is a bunch of string > objects and I start taking it apart with [H|T] = List.. > > When processing the last string in the list I end up taking apart the > individual characters of the string. If I do a type-check it tells me > it is a list. > > I usually have to do a work around to handle this. If it was a type I > would easily know when I am done with the list. The various pros and cons of having a special string type in Erlang have been discussed before in this mailing list; please refer to the archives of the list for finding them. The arguments for a string data type are often more involved than yours and typically are centered around space concerns. If your only grief with strings-as-lists is the above "programming confusion", may I suggest that you wrap your strings inside a tuple-pair with a 'str' tag? I.e., instead of a list of strings ["hello", "world"] you process lists of the form [{str,"hello"}, {str,"world"}]. When you become more comfortable with list processing in general, you can then drop the 'str' tuple wrappers from your code, and most probably you will also realize that the list(string()) representation is not so bad after all. Kostis From g@REDACTED Sat Jun 23 19:19:35 2018 From: g@REDACTED (Guilherme Andrade) Date: Sat, 23 Jun 2018 18:19:35 +0100 Subject: [erlang-questions] [ANN] stacktrace_compat: get_stacktrace() compatibility on OTP 21 Message-ID: Hello list, I?m pleased to announce the release of stacktrace_compat 1.0, a workaround for the deprecation of erlang:get_stacktrace() on Erlang/OTP 21. It intends on smoothing near-future maintenance of projects that are to support both pre- and post-deprecation code by avoiding code duplication or ungainly macros. It replaces calls to :get_stacktrace() with references to bindings (whether explicit or implicit) that capture the stacktrace using the new syntax. * Overview: https://github.com/g-andrade/stacktrace_compat/blob/master/README.md * Hex.pm package: https://hex.pm/packages/stacktrace_compat -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From heinz@REDACTED Sun Jun 24 00:20:19 2018 From: heinz@REDACTED (heinz@REDACTED) Date: Sat, 23 Jun 2018 18:20:19 -0400 Subject: [erlang-questions] [ANN] stacktrace_compat: get_stacktrace() compatibility on OTP 21 In-Reply-To: References: Message-ID: <032e01d40b40$60a70cc0$21f52640$@licenser.net> That?s great stuff thanks! From: erlang-questions-bounces@REDACTED On Behalf Of Guilherme Andrade Sent: Saturday, June 23, 2018 13:20 To: Erlang-Questions Questions Subject: [erlang-questions] [ANN] stacktrace_compat: get_stacktrace() compatibility on OTP 21 Hello list, I?m pleased to announce the release of stacktrace_compat 1.0, a workaround for the deprecation of erlang:get_stacktrace() on Erlang/OTP 21. It intends on smoothing near-future maintenance of projects that are to support both pre- and post-deprecation code by avoiding code duplication or ungainly macros. It replaces calls to :get_stacktrace() with references to bindings (whether explicit or implicit) that capture the stacktrace using the new syntax. * Overview: https://github.com/g-andrade/stacktrace_compat/blob/master/README.md * Hex.pm package: https://hex.pm/packages/stacktrace_compat -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Sun Jun 24 12:41:52 2018 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Sun, 24 Jun 2018 12:41:52 +0200 Subject: [erlang-questions] gen_statem: Join states? In-Reply-To: References: Message-ID: Hello, Jesper. My problem is not how to evaluate the information whether events have arrived. I simply kept a list of IDs and used lists:delete/2 and checked for the empty list. Your solution however also looks very useful. My main problem was _where_ to stop this information. gen_statem gives you a possibility to store state, but then you would have to store information about the join (the list of events that have not arrived yet - or your map of events that do have arrived yet) together with the actual data pertaining to processing the events themselves. (For example we store in this other data the contents of our parsed configuration.) So, if I simply store the join information there, I have to create a data structure differentiating the two - like a record or a map. This would then require updating it every time and I did not consider it a very generic solution. I considered doing something like { [ ], OtherData } but I didn't particularly like this situation - it still mixes information about the flow with the other data. That's why I looked for a solution within the FSM itself which I initially didn't find but now have Frans' pointers to follow. Complex states with gen_statem:handle_event/4 seem to be a good choice - like a { waitForJoinEvents, [ ]} kind of state. I guess this could be generalized especially by using functions like you decribed. At least the "other data" would remain clearly separate from the data mean to direct flow. Functions could be written to evaluate this { StateName, EventList } tuple together with the incoming message and make decisions when for example the state should be changed. (Trivially, even the StateName can be ignored, though, as it's mostly useful for having a handle_event/4 that calls the the join with the right follow-up action when the join is satisfied.) I guess I simply have to rewrite the involved FSMs as handle_event_function ones. Given the greater flexibility of the approach it seems to be the better default choice. Thank you and kind regards, Oliver ? ? Gesendet:?Samstag, 23. Juni 2018 um 12:21 Uhr Von:?"Jesper Louis Andersen" An:?"Oliver Korpilla" Cc:?"Erlang (E-mail)" Betreff:?Re: [erlang-questions] gen_statem: Join states? I usually stuff the events into a map and then have a handle_continue/2 like behavior to check if everything in the map is present. ? Simplified: ? event(a, State) -> ? ? check(State#{ a => true }); event(b, State) -> ? ? check(State#{ b => true }); ... ? check(#{ a := A, b := B} = State) -> ? ? end(A, B, State); check(#{} = State) -> ? ? {noreply, ...}. ? Usually, rather than storing true, I'd store the event data there, and rather than having a map() at the top-level I'd probably have a record. But the gist of the idea is there. A more advanced variant of this will have a defined state machine and valid transitions between them. Then check is also an invariant on the transitions validity and it becomes a (runtime) contract to check against. ? Aside: Idris can do this check at compile time, but since we don't have dependent types... ? ?? On Sat, Jun 23, 2018 at 9:07 AM Oliver Korpilla wrote:Hello. In the system we're building we're using gen_statem processes to keep track of signalling scenarios, essentially series of different types of messages from various sources. For linear or branching scenarios gen_statem is well prepared. But there's one thing that seemed to be missing and that is support for joins? I recently reviewed a developer's code that implemented receiving two different messages that could arrive in any order as four state functions - receiving A first (1), then receiving B (2), then entering an end state; or receiving B first (3), then receiving A (4), then entering an end state. The code entering the end state was duplicated in both (2) and (4). What essentially was needed was a "check list" of which messages had already arrived to determine the transition into the end state. Such "check list" scenarios are quite common in our work (and can grow to be more complex) and I couldn't find a direct support for it (though I was limiting myself to state_function mode), so we're now starting a separate "join process" - a gen_server that keeps and maintain the join list - and do the decision now from a single wait state. Given how light-weight Erlang processes are this works fine. What prevented me from doing this through data maintained between calls was that I would have to partition the state into information for tracking the join and the actual information kept to process the signals. This seemed less generic and clean. I'm also aware that had I rewritten the whole state machine to handle_event functions and utilized a complex state this could have helped achieve the same. What is essentially is needed, from my POV, is a way to track information pertaining to state transitions separate from information for processing incoming events, and complex states seem to do that...? What is your preferred way of doing this? Is it possible to somehow support this at the library level? Are handle_event/complex states the way to go in such situations? Thank you and regards, Oliver _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED[mailto:erlang-questions@REDACTED] http://erlang.org/mailman/listinfo/erlang-questions? ?-- J. From connolly.damien@REDACTED Sun Jun 24 12:36:40 2018 From: connolly.damien@REDACTED (Damien Towning) Date: Sun, 24 Jun 2018 20:36:40 +1000 Subject: [erlang-questions] More elegant way to Message-ID: Hi everyone, I have some c code using c_node that I have successfully got working with Erlang ( through Elixir ) and I am wondering if on the C side a more elegant way exists to pass data back to the Erlang/Elixir than what I am currently doing. My C code looks a bit like this. else if (erl_match(msg_pattern_b, emsg->msg)) { ETERM *term = erl_var_content(msg_pattern_b, "Term"); response = erl_format("{ok, ~w}",erl_mk_string(floats_as_string())); erl_free_term(term); } I've parsed a series of floats in to a char * and then passed that back to erl_mk_string(). I hope I have framed this question in a way that makes sense. I'd like to take a list of floating point values directly and pass it directly back to my calling Elixir code as a list without having to parse it in and out of a string. Is this possible. I have spent quite a bit of time now digging through the erl_ documentation and trying to find an example but I just can't quite see it. Thanks ! -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathaniel@REDACTED Sun Jun 24 19:58:22 2018 From: nathaniel@REDACTED (Nathaniel Waisbrot) Date: Sun, 24 Jun 2018 13:58:22 -0400 Subject: [erlang-questions] More elegant way to In-Reply-To: References: Message-ID: <34538AA3-98D4-4677-918D-82ACD8850C32@waisbrot.net> > else if (erl_match(msg_pattern_b, emsg->msg)) { > ETERM *term = erl_var_content(msg_pattern_b, "Term"); > response = erl_format("{ok, ~w}",erl_mk_string(floats_as_string())); > erl_free_term(term); > } > Your sample code looks either buggy or incomplete (you create `term` and then call `erl_free_term` on it without ever using it). If understand your question, you want to return (e.g.) `{ok, [1.0, 1.1, 1.2, 2.0]}`. Right now you're returning something like `{ok, "[1.0, 1.1, 1.2, 2.0]"}` and then parsing the values out of the char-list in BEAM-land? How about (untested) ``` ETERM* floats_to_list(float clist[], size_t length) { ETERM* elist[length]; unsigned i; for (i = 0; i < length; i++) { elist[i] = erl_mk_float(clist[i]); } return erl_mk_list(elist, length); } void top_function() { float *list_of_floats; size_t size_of_list; ETERM *response; // ... // somehow get these values loaded // ... response = floats_to_list(list_of_floats, size_of_list); send_response(response); // ... } ``` From rickard@REDACTED Mon Jun 25 02:42:09 2018 From: rickard@REDACTED (Rickard Green) Date: Mon, 25 Jun 2018 02:42:09 +0200 Subject: [erlang-questions] inet:getstat on non-socket crashes the VM? In-Reply-To: References: Message-ID: On Fri, Jun 22, 2018 at 1:53 PM, Bekes, Andras G < Andras.Bekes@REDACTED> wrote: > > Hi Erlangers, > > > > I noticed that calling inet:getstat/1 is a dangerous operation, as it can kill the whole VM if used inappropriately. > > > > It is easy to reproduce: > > > > Eshell V9.3 (abort with ^G) > > 1> application:start(sasl). > > ok > > 2> [inet:getstat(X) || X<-erlang:ports()]. > > Failed to write to erl_child_setup: 9 > > Crash dump is being written to: erl_crash.dump...done > > > > Without sasl, it doesn?t kill the VM, but the terminal gets into a faulty state and further work is not possible. > > I know calling it on a non-socket is an erroneous action, but crashing the VM doesn?t seem to be the most appropriate consequence. > > What do you think? > > > > Regards, > > > > Andras G. Bekes > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > Broadcasting random stuff around is dangerous. Different ports (and processes) have different protocols that they expect to be used. A message that means something for one port may mean something completely different for another port. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders.gs.svensson@REDACTED Mon Jun 25 10:01:24 2018 From: anders.gs.svensson@REDACTED (anders.gs.svensson@REDACTED) Date: Mon, 25 Jun 2018 10:01:24 +0200 Subject: [erlang-questions] erlang-questions Digest, Vol 378, Issue 6 In-Reply-To: References: Message-ID: <23344.41300.304739.925702@gargle.gargle.HOWL> rveyland2@REDACTED writes: > > Dear Erlang Community, > > I?m a beginner in Erlang, and i have a question on diameter stack behavior. > > I ?m working on a sw acting as diameter peer server with a S6A app. > > I have noticed that when an diameter request is received with an error (for example an additional AVP not supported by dict), even if the packet provided in return value of the handle_request call back contains a valid answer with an AVP result-code=Success and header with errors=[] and is_error=false, the diameter stack return to client peer a diameter packet with an error 5001 (unsupported_avp) which is corresponding to the initial error found on request decoding (and not the answer sent in {reply, Reply}. The diameter application can indeed modify your answer, but you can tell it not to. Here's the doc for the handle_request callback in diameter_app(3): The errors field specifies any results codes identifying errors found while decoding the request. This is used to set Result-Code and/or Failed-AVP in a returned answer unless the callback returns a #diameter_packet{} whose errors field is set to either a non-empty list of its own, in which case this list is used instead, or the atom false to disable any setting of Result-Code and Failed-AVP. Note that the errors detected by diameter are of the 3xxx and 5xxx series, Protocol Errors and Permanent Failures respectively. The errors list is empty if the request has been received in the relay application. That is set the errors field to false instead of [] and you should get what you're expecting. (As Chandru did in his example.) > If the initial diameter request is valid verus dict, then the answer provided by callback is sent back to client peer as expected. > > I would like to confirm that it is due to a bug from my sw (most probably?? ) > > So I have some questions?: > - Is it a normal of diameter stack behavior Yes. Far enough back in time this was the only possible behaviour, but in more recent times you can ... > o if yes how to avoid/bypass it?? (how to send a valid answer even if request is seen in error) ... set errors = false, as above. > o If not?: on which criteria a predefined answer is sent back to the transport layer instead of the application?s callback one?? > - More globally: What is the simplest way to debug diameter stack in order to verify that data sent back by callback handler are ??as expected???? Like Chandru said, tracing on diameter_codec lets you see what is actually sent. Something like the following will do in this case. dbg:tracer(). dbg:p(all, c). dbg:tp(diameter_codec, encode, x). Or [] instead of x if you only want to see what's encoded. /Anders, Erlang/OTP > > Thx a lot by advance for your support. > > BR > > Ren? Veyland From v@REDACTED Mon Jun 25 12:50:36 2018 From: v@REDACTED (Valentin Micic) Date: Mon, 25 Jun 2018 12:50:36 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: <00B3E40C-06B5-4FA0-9757-43D3A2640DF2@pharos-corp.com> On 20 Jun 2018, at 7:06 PM, Lukas Larsson wrote: > On Wed, Jun 20, 2018 at 1:56 PM Jonas Falkevik wrote: > > Are you adding the multicast network to the loop back interface? Using some other interface that does not allow multicast traffic? > > > I though I had, but apparently not.... I managed to reproduce the test case now. > > I also spent some time staring at the code in the inet_driver and realized what the problem was in there. It would seem that a performance/feature fix for SCTP in R15B inadvertently introduced this behaviour for UDP :( I've pushed a bug fix to that problem in the inet_driver to the same branch. In my tests the scheduler utilization goes form 71% to about 4% with both patches. With the fix in the inet driver, my allocator patch does not really do any difference for this specific testcase, but I'll keep that fix anyways as it is a "good thing". I should probably also add a benchmark for this so that it does not happen again.... > > I'd also like to add that changing the test-case to use a smaller user-space buffer also has the same effect. So if anyone is running a performance critical UDP server that has set the "recbuf" to a high value, I really recommend that you lower the "buffer" size to something close to the max expected packet size. In the example that Danil provided I applied the following patch: > > diff --git a/udptest.erl b/udptest.erl > index 16a1798..4edeef0 100644 > --- a/udptest.erl > +++ b/udptest.erl > @@ -33,7 +33,9 @@ send_packet(ID, S, Addr, Port) -> > > start_reader(Addr, Port) -> > GwIP = {0,0,0,0}, % {127,0,0,1}, > - Common = [binary,{reuseaddr,true},{recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], > + Common = [binary,{reuseaddr,true}, > + {buffer,1500}, %% 1500 is just an example value, don't just copy this. You need to know what you max UDP packet size will be. > + {recbuf,2*1024*1024},inet,{read_packets,100},{active,500}], > Multicast = [{multicast_ttl,4},{multicast_loop,true},{ip,Addr},{add_membership,{Addr,GwIP}}], > Options = Common ++ Multicast, > spawn(fun() -> run_reader(Port, Options) end). > > and the scheduler utilization dropped to about 4% there as well. > > Lukas > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Juts one observation about inet:setopts/2 which may not be present in R21, but certainly was in earlier releases. When setting the "user buffer", the order of attributes specification seems to matter, e.g. inet:setopts( S, [{buffer, 512}, {recbuf, 16#1ffff}, {sndbuf, 16#1ffff}]). will set the buffer to 512, whilst: inet:setopts( S, [{recbuf, 16#1ffff}, {sndbuf, 16#1ffff}, {buffer, 512}]). will leave it as 16#1ffff! (e.g. set to recbuf size). Kind regards V/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From v@REDACTED Mon Jun 25 13:13:52 2018 From: v@REDACTED (Valentin Micic) Date: Mon, 25 Jun 2018 13:13:52 +0200 Subject: [erlang-questions] UDP receive performance In-Reply-To: References: <20180524072357.GA97377@erix.ericsson.se> <355E67A6-9883-451D-A31D-8CC710244719@gmail.com> <19DDAEE8-037B-4793-9C83-7D8D5C73C234@mobilearts.com> <0AE32206-3A53-404A-A97E-426E0E974FA1@mobilearts.com> Message-ID: On 22 Jun 2018, at 10:27 AM, Lukas Larsson wrote: > On Fri, 22 Jun 2018, 09:24 pablo platt, wrote: > Does this bug only affect multicast UDP or also normal UDP? > > It effects reception of all UDP messages. > > > What values should I use for buffer and recbuf in a UDP socket receiving 1Mbps with 1500 Bytes MTU? > I thought I need N*MTU recbuf so the pid will be able to handle a small burst of UDP packets. > > There is no need to set buffer larger than your MTU. The recbuf however should be large enough to handle any bursts that may happen. > > This is what I currently have: > gen_udp:open(0, [binary, {active, once}, {recbuf, 16*1024}]) > > I doubt that you will see any large performance differences by setting the buffer size to, let's say 2*1024. But it will as always depend on your application. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions A high volume testing we have performed using earlier versions (R15+), yielded far less (or rather none) packet drops when recbuf has been set to a higher value. Cannot recall what were the hardware specs, but I remember that without adjusting recbuf, the packet drops were experienced at the rate that was about 10,000 packet/sec. With adjustment for recbuf (arbitrarily set to 1MB), we were able to push 70,000 packets/sec without a drop throughout tests. The testing was not performed to measure impact on CPU, but rather to establish that packet drops were function of recbuf size (and/or sender's sndbuf size). We have concluded this to be the case for the packet sizes not exceeding MTU. Kind regards V/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Mon Jun 25 15:16:10 2018 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Mon, 25 Jun 2018 15:16:10 +0200 Subject: [erlang-questions] Patch package OTP 20.3.8.1 released Message-ID: Patch Package: OTP 20.3.8.1 Git Tag: OTP-20.3.8.1 Date: 2018-06-25 Trouble Report Id: OTP-15098, OTP-15120, OTP-15148, OTP-15149 Seq num: ERIERL-177, ERIERL-192, ERIERL-194, ERIERL-199 System: OTP Release: 20 Application: inets-6.5.2.1, ssh-4.6.9.1, syntax_tools-2.1.4.1 Predecessor: OTP 20.3.8 Check out the git tag OTP-20.3.8.1, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- inets-6.5.2.1 --------------------------------------------------- --------------------------------------------------------------------- The inets-6.5.2.1 application can be applied independently of other applications on a full OTP 20 installation. --- Improvements and New Features --- OTP-15120 Application(s): inets Related Id(s): ERIERL-192 Options added for setting low-level properties on the underlying TCP connections. The options are: sock_ctrl, sock_data_act and sock_data_pass. See the manual for details. Full runtime dependencies of inets-6.5.2.1: erts-6.0, kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, ssl-5.3.4, stdlib-2.0 --------------------------------------------------------------------- --- ssh-4.6.9.1 ----------------------------------------------------- --------------------------------------------------------------------- Note! The ssh-4.6.9.1 application can *not* be applied independently of other applications on an arbitrary OTP 20 installation. On a full OTP 20 installation, also the following runtime dependencies have to be satisfied: -- crypto-4.2 (first satisfied in OTP 20.2) -- public_key-1.5.2 (first satisfied in OTP 20.2) --- Fixed Bugs and Malfunctions --- OTP-15148 Application(s): ssh Related Id(s): ERIERL-194 SFTP clients reported the error reason "" if a non-OTP sftp server was killed during a long file transmission. Now the signal name (for example "KILL") will be the error reason if the server's reason is empty. The documentation also lacked type information about this class of errors. OTP-15149 Application(s): ssh Related Id(s): ERIERL-199 Fix ssh_sftp decode error for sftp protocol version 4 Full runtime dependencies of ssh-4.6.9.1: crypto-4.2, erts-6.0, kernel-3.0, public_key-1.5.2, stdlib-3.3 --------------------------------------------------------------------- --- syntax_tools-2.1.4.1 -------------------------------------------- --------------------------------------------------------------------- The syntax_tools-2.1.4.1 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-15098 Application(s): syntax_tools Related Id(s): ERIERL-177 Fix a bug regarding reverting map types. Full runtime dependencies of syntax_tools-2.1.4.1: compiler-7.0, erts-9.0, kernel-5.0, stdlib-3.4 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4161 bytes Desc: S/MIME Cryptographic Signature URL: From roger@REDACTED Mon Jun 25 16:28:24 2018 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 25 Jun 2018 15:28:24 +0100 Subject: [erlang-questions] httpc broke URL parsing in OTP-21.0? Message-ID: 20.3... 1> inets:start(). ok 2> httpc:request(get, {"http://localhost:31316/EluoueLg-ueL/?x=32&y=54&z[0]=1&z[1]=2&q[]=5&q[]=4&p=9&p=8", []}, [], []). {error,{failed_connect,[{to_address,{"localhost",31316}}, {inet,[inet],econnrefused}]}} ...fair enough (no server is running). 21.0... 1> inets:start(). ok 2> httpc:request(get, {"http://localhost:31316/EluoueLg-ueL/?x=32&y=54&z[0]=1&z[1]=2&q[]=5&q[]=4&p=9&p=8", []}, [], []). ** exception error: no function clause matching uri_string:parse({error,invalid_uri,":"}) (uri_string.erl, line 337) in function httpc:request/5 (httpc.erl, line 179) Not good :( Cheers, Roger. From roger@REDACTED Mon Jun 25 16:31:15 2018 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 25 Jun 2018 15:31:15 +0100 Subject: [erlang-questions] httpc broke URL parsing in OTP-21.0? In-Reply-To: References: Message-ID: Pared it down: 17> httpc:request(get, {"http://localhost/?x[0]=1", []}, [], []). ** exception error: no function clause matching uri_string:parse({error,invalid_uri,":"}) (uri_string.erl, line 337) in function httpc:request/5 (httpc.erl, line 179) 18> httpc:request(get, {"http://localhost/?x=1", []}, [], []). {ok,{{"HTTP/1.1",200,"OK"}, ...} It doesn't seem to like the square brackets. On 25 June 2018 at 15:28, Roger Lipscombe wrote: > 20.3... > > 1> inets:start(). > ok > 2> httpc:request(get, > {"http://localhost:31316/EluoueLg-ueL/?x=32&y=54&z[0]=1&z[1]=2&q[]=5&q[]=4&p=9&p=8", > []}, [], []). > {error,{failed_connect,[{to_address,{"localhost",31316}}, > {inet,[inet],econnrefused}]}} > > ...fair enough (no server is running). > > 21.0... > > 1> inets:start(). > ok > 2> httpc:request(get, > {"http://localhost:31316/EluoueLg-ueL/?x=32&y=54&z[0]=1&z[1]=2&q[]=5&q[]=4&p=9&p=8", > []}, [], []). > ** exception error: no function clause matching > uri_string:parse({error,invalid_uri,":"}) (uri_string.erl, line 337) > in function httpc:request/5 (httpc.erl, line 179) > > Not good :( > > Cheers, > Roger. From rveyland2@REDACTED Mon Jun 25 17:30:28 2018 From: rveyland2@REDACTED (=?utf-8?Q?Ren=C3=A9_Veyland_Mobile?=) Date: Mon, 25 Jun 2018 17:30:28 +0200 Subject: [erlang-questions] erlang-questions Digest, Vol 378, Issue 6 In-Reply-To: <23344.41300.304739.925702@gargle.gargle.HOWL> References: <23344.41300.304739.925702@gargle.gargle.HOWL> Message-ID: <5b310a95.1c69fb81.d2607.27b3@mx.google.com> Hi Anders I will try with error=false instead of [] as it was coded. I wrongly understood the text. I understood that the stack returns an error excepted if error field was set to [] or false. My fault. Thx for feedback. Br Ren? Provenance?: Courrier pour Windows 10 De?: anders.gs.svensson@REDACTED Envoy? le?:lundi 25 juin 2018 10:01 ??: erlang-questions@REDACTED Cc?: rveyland2@REDACTED; chandrashekhar.mullaparthi@REDACTED Objet?:Re: erlang-questions Digest, Vol 378, Issue 6 rveyland2@REDACTED writes: > > Dear Erlang Community, > > I?m a beginner in Erlang, and i have a question on diameter stack behavior. > > I ?m working on a sw acting as diameter peer server with a S6A app. > > I have noticed that when an diameter request is received with an error (for example an additional AVP not supported by dict), even if the packet provided in return value of the handle_request call back contains a valid answer with an AVP result-code=Success and header with errors=[] and is_error=false, the diameter stack return to client peer a diameter packet with an error 5001 (unsupported_avp) which is corresponding to the initial error found on request decoding (and not the answer sent in {reply, Reply}. The diameter application can indeed modify your answer, but you can tell it not to. Here's the doc for the handle_request callback in diameter_app(3): The errors field specifies any results codes identifying errors found while decoding the request. This is used to set Result-Code and/or Failed-AVP in a returned answer unless the callback returns a #diameter_packet{} whose errors field is set to either a non-empty list of its own, in which case this list is used instead, or the atom false to disable any setting of Result-Code and Failed-AVP. Note that the errors detected by diameter are of the 3xxx and 5xxx series, Protocol Errors and Permanent Failures respectively. The errors list is empty if the request has been received in the relay application. That is set the errors field to false instead of [] and you should get what you're expecting. (As Chandru did in his example.) > If the initial diameter request is valid verus dict, then the answer provided by callback is sent back to client peer as expected. > > I would like to confirm that it is due to a bug from my sw (most probably?? ) > > So I have some questions?: > - Is it a normal of diameter stack behavior Yes. Far enough back in time this was the only possible behaviour, but in more recent times you can ... > o if yes how to avoid/bypass it?? (how to send a valid answer even if request is seen in error) ... set errors = false, as above. > o If not?: on which criteria a predefined answer is sent back to the transport layer instead of the application?s callback one?? > - More globally: What is the simplest way to debug diameter stack in order to verify that data sent back by callback handler are ??as expected???? Like Chandru said, tracing on diameter_codec lets you see what is actually sent. Something like the following will do in this case. dbg:tracer(). dbg:p(all, c). dbg:tp(diameter_codec, encode, x). Or [] instead of x if you only want to see what's encoded. /Anders, Erlang/OTP > > Thx a lot by advance for your support. > > BR > > Ren? Veyland -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila.r.nohl@REDACTED Mon Jun 25 18:00:08 2018 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Mon, 25 Jun 2018 18:00:08 +0200 Subject: [erlang-questions] inet:getstat on non-socket crashes the VM? In-Reply-To: References: Message-ID: 2018-06-22 13:53 GMT+02:00 Bekes, Andras G : > Hi Erlangers, > > > > I noticed that calling inet:getstat/1 is a dangerous operation, as it can > kill the whole VM if used inappropriately. > > > > It is easy to reproduce: > > > > Eshell V9.3 (abort with ^G) > > 1> application:start(sasl). > > ok > > 2> [inet:getstat(X) || X<-erlang:ports()]. > > Failed to write to erl_child_setup: 9 > > Crash dump is being written to: erl_crash.dump...done > > > > Without sasl, it doesn?t kill the VM, but the terminal gets into a faulty > state and further work is not possible. > > I know calling it on a non-socket is an erroneous action, but crashing the > VM doesn?t seem to be the most appropriate consequence. > > What do you think? I raised a ticket about a similar issue a couple of months ago on bugs.erlang.org, but got similar answer as you. I cannot link the ticket number as all I get on bugs.erlang.org is this: type Exception report message Unable to establish a connection with the database. (Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.) description The server encountered an internal error that prevented it from fulfilling this request. exception com.opensymphony.module.propertyset.PropertyImplementationException: Unable to establish a connection with the database. (Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.) From andreas.schultz@REDACTED Mon Jun 25 18:05:32 2018 From: andreas.schultz@REDACTED (Andreas Schultz) Date: Mon, 25 Jun 2018 18:05:32 +0200 Subject: [erlang-questions] httpc broke URL parsing in OTP-21.0? In-Reply-To: References: Message-ID: Square brackets are not permitted in the query part of an URL. You need to percent encode them (see https://tools.ietf.org/html/rfc3986#section-3.4). 6> httpc:request(get, {"http://localhost:8080/?x%5B0%5D=1", []}, [], []). {ok,{{"HTTP/1.1",200,"OK"},...} However, many browsers and web servers will accept square brackets in the URI. And some (probably many) broken clients will not encode square brackets in requests. e.g.: $ wget --debug -O /dev/null -S 'http://localhost:8080/?x [0]=1' ... ---request begin--- GET /?x%20[0]=1 HTTP/1.1 Space is encoded, brackets are not... Erlang will also (rightly) complain about spaces: 7> httpc:request(get, {"http://localhost:8080/?x =1", []}, [], []). ** exception error: no function clause matching uri_string:parse({error,invalid_uri,":"}) (uri_string.erl, line 337) in function httpc:request/5 (httpc.erl, line 179) Andreas Roger Lipscombe schrieb am Mo., 25. Juni 2018 um 16:31 Uhr: > Pared it down: > > 17> httpc:request(get, {"http://localhost/?x[0]=1", []}, [], []). > ** exception error: no function clause matching > uri_string:parse({error,invalid_uri,":"}) (uri_string.erl, line 337) > in function httpc:request/5 (httpc.erl, line 179) > 18> httpc:request(get, {"http://localhost/?x=1", []}, [], []). > {ok,{{"HTTP/1.1",200,"OK"}, ...} > > It doesn't seem to like the square brackets. > > On 25 June 2018 at 15:28, Roger Lipscombe wrote: > > 20.3... > > > > 1> inets:start(). > > ok > > 2> httpc:request(get, > > {" > http://localhost:31316/EluoueLg-ueL/?x=32&y=54&z[0]=1&z[1]=2&q[]=5&q[]=4&p=9&p=8 > ", > > []}, [], []). > > {error,{failed_connect,[{to_address,{"localhost",31316}}, > > {inet,[inet],econnrefused}]}} > > > > ...fair enough (no server is running). > > > > 21.0... > > > > 1> inets:start(). > > ok > > 2> httpc:request(get, > > {" > http://localhost:31316/EluoueLg-ueL/?x=32&y=54&z[0]=1&z[1]=2&q[]=5&q[]=4&p=9&p=8 > ", > > []}, [], []). > > ** exception error: no function clause matching > > uri_string:parse({error,invalid_uri,":"}) (uri_string.erl, line 337) > > in function httpc:request/5 (httpc.erl, line 179) > > > > Not good :( > > > > Cheers, > > Roger. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- -- Dipl.-Inform. Andreas Schultz ----------------------- enabling your networks ---------------------- Travelping GmbH Phone: +49-391-81 90 99 0 Roentgenstr. 13 Fax: +49-391-81 90 99 299 39108 Magdeburg Email: info@REDACTED GERMANY Web: http://www.travelping.com Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 --------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Mon Jun 25 19:04:03 2018 From: roger@REDACTED (Roger Lipscombe) Date: Mon, 25 Jun 2018 18:04:03 +0100 Subject: [erlang-questions] httpc broke URL parsing in OTP-21.0? In-Reply-To: References: Message-ID: On 25 June 2018 at 17:05, Andreas Schultz wrote: > Square brackets are not permitted in the query part of an URL. You need to > percent encode them (see https://tools.ietf.org/html/rfc3986#section-3.4). Sure, but this is a breaking change. Source that did work now doesn't. > However, many browsers and web servers will accept square brackets in the > URI. And some (probably many) broken clients will not encode square brackets > in requests. Chrome passes the square brackets straight through if you type them into the address bar; haven't checked any other browsers (yet). > $ wget --debug -O /dev/null -S 'http://localhost:8080/?x [0]=1' curl (provided you specify --globoff; and, tbf, it *does* have a warning in the manpage) also passes them straight through. I've fixed my code -- it was just one line -- but ... breaking change. From per@REDACTED Mon Jun 25 22:30:32 2018 From: per@REDACTED (Per Hedeland) Date: Mon, 25 Jun 2018 22:30:32 +0200 Subject: [erlang-questions] inet:getstat on non-socket crashes the VM? In-Reply-To: References: Message-ID: On 2018-06-25 02:42, Rickard Green wrote: > > > On Fri, Jun 22, 2018 at 1:53 PM, Bekes, Andras G > wrote: > > > > Hi Erlangers, > > > > > > > > I noticed that calling inet:getstat/1 is a dangerous operation, as it can kill the whole VM if used inappropriately. > > > > > > > > It is easy to reproduce: > > > > > > > > Eshell V9.3 (abort with ^G) > > > > 1> application:start(sasl). > > > > ok > > > > 2> [inet:getstat(X) || X<-erlang:ports()]. > > > > Failed to write to erl_child_setup: 9 > > > > Crash dump is being written to: erl_crash.dump...done > > > > > > > > Without sasl, it doesnt kill the VM, but the terminal gets into a faulty state and further work is not possible. > > > > I know calling it on a non-socket is an erroneous action, but crashing the VM doesnt seem to be the most appropriate consequence. > > > > What do you think? > > > > > > > > Regards, > > > > > > > > Andras G. Bekes > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > Broadcasting random stuff around is dangerous. Different ports (and processes) have different protocols that they expect to be used. A message that means something for one port may mean something > completely different for another port. Agreed in principle - inet:getstat/1 and many other socket operations (I forget which one we ran into this problem with) end up calling erlang:port_control/3, where the 'Operation' argument is indeed an arbitrary integer with semantics that are specific to the driver / port program referenced by the 'Port' argument - BUT: a) Programming errors do happen, and IMHO intentionally terminating the VM as a means of error handling (which is what the "forker" driver does in this case) is really not acceptable for a system with all the sophisticated error handling and expected robustness of Erlang/OTP, and b) it's a reasonable expectation that *all* drivers and port programs, and specifically the drivers and port programs included in the Erlang/OTP distribution, at least do a basic validation of the 'Operation' argument, and I would even suggest that c) it would be reasonable for the drivers and port programs included in the Erlang/OTP distribution to use unique values for the 'Operation' argument, and even have a documented range of those values, to make it possible for user-written drivers and port programs to avoid using them. Anyway, in this particular case, the patch below (for OTP 20.3.6), which just addresses b) for the "forker" driver, but thereby in most cases also avoids a), seems like a good idea to me... (AFAIR, prim_inet will actually turn the 'badarg' into 'einval', but it's still way better than the driver terminating the VM after failing to use some random and potentially un-initialized arguments to do what it *thought* was requested...) --Per Hedeland --- a/erts/emulator/sys/unix/sys_drivers.c +++ b/erts/emulator/sys/unix/sys_drivers.c @@ -1851,6 +1851,11 @@ static ErlDrvSSizeT forker_control(ErlDrvData e, unsigned int cmd, char *buf, ErlDrvPort port_num = (ErlDrvPort)e; int res; + /* fail with badarg if wrong port_control call */ + if (cmd != 'S') { + return -1; + } + driver_enq(port_num, buf, len); if (driver_sizeq(port_num) > sizeof(*proto)) { return 0; From soverdor@REDACTED Mon Jun 25 23:12:47 2018 From: soverdor@REDACTED (Sam Overdorf) Date: Mon, 25 Jun 2018 14:12:47 -0700 Subject: [erlang-questions] String type In-Reply-To: References: Message-ID: Someone gave me a link to a previous discussion of this and it's problems. I read it and decided that I need to change my process and not modify Erlang. Thanks for the response, Sam On Mon, Jun 25, 2018 at 7:30 AM, Richard O'Keefe wrote: > Yes, people have often considered adding a "real" string > data type to Erlang. With the move to 64-bit machines > this became even more interesting. However, in a Unicode > world, it really is not clear what string *is*. > > For example, in the old ASCII days, it was clear that a > string was a sequence of characters, and all characters > were the same size, and the actual ASCII definition made > it clear that NUL and DEL probably should NOT be allowed > in a string. US programmers (hence US textbooks, hence > practically everyone in the English-speaking world) > quietly ignored the fact that the ASCII standard explicitly > allowed overstrikes so that you could get u-with-umlaut > by doing <"> or even <"> . So in fact in > ASCII a "character" could well be a sequence of code-points > and that is in fact why ` and ^ are in the ASCII set, and > it wasn't therefore *true* that all characters were the > same size. > > In the ISO 8851 family, the standardisers bowed to reality > and forbade overstriking, introducing precomposed accented > letters instead. So the statement that ASCII is a subset > of ISO 8859/1 is a half truth: the codepoints are a subset > but ASCII allows you to DO things with them that Latin-1 > does not. > > Unicode has it both ways. It has precomposed characters > like u-with-umlaut, and it also has composed characters > like u-followed-by-(floating umlaut). Which means we > now have to ask "is a string a sequence of codepoints > or a sequence of characters". But it's more complicated. > See Unicode Technical Annex 29 "Unicode Text Segmentation" > for the horrible details. But the alternatives are > > - sequence of bytes (in UTF8) > - sequence of 16-bit units (UTF16) > - sequence of code-points > + sequence of legacy grapheme clusters > + sequence of extended grapheme clusters > + sequence of tailored grapheme clusters > bearing in mind that > * some code points are always illegal > * most code points are unassigned > * some sequences of code points are illegal > * in particular, legal sequences may have > illegal subsequences, so the "substring" > operation is problematic. > > Let's not even try to think about the existence > of multiple characters with identical appearance, > multiple ways to encode many characters, > invisible characters, characters forbidden by design > then introduced then deprecated, and the question > of whether control marks like redundant direction > indicators should count in deciding whether strings > are equal. > > If you are dealing with text where you are actually > looking at the characters doing some sort of parsing, > the chances are you want a list of tokens or even > some sort of tree rather than a string. > > I'm actually more interested in the fact that you say > you have trouble with lists of strings. Can you > provide an example of the kind of code you have > trouble with? If you use the Dialyzer, it has no > trouble expressing the difference between a list of > integers and a list of lists of integers, and even > without it, it's not a commonly reported problem. > > For example, suppose we have a list of strings and > want to paste them together with spaces between > them. This is called "unwords" in Haskell. Let's > start with the Haskell version. > > unwords :: [String] -> String > unwords [] = [] > unwords (w:ws) = w ++ aux ws > where aux [] = [] > aux (y:ys) = " " ++ y ++ aux ys > > Let's put that into Erlang: > > unwords([]) -> []; > unwords([W|Ws]) -> W ++ unwords_aux(Ws). > > unwords_aux([]) -> ""; > unwords_aux([Y|Ys]) -> " " ++ Y ++ unwords_aux(Ys). > > By the way, this kind of thing is spectacularly > inefficient in languages like Java, which is why Java > has StringBuilder as well as String. This is one of > many reasons why I have a slogan STRINGS ARE WRONG. > > > > On 23 June 2018 at 09:41, Sam Overdorf wrote: >> >> Has anyone considered making string a type and not a list of chars. >> >> I seem to have a lot of trouble when a list is a bunch of string >> objects and I start taking it apart with [H|T] = List.. >> >> When processing the last string in the list I end up taking apart the >> individual characters of the string. If I do a type-check it tells me >> it is a list. >> >> I usually have to do a work around to handle this. If it was a type I >> would easily know when I am done with the list. >> >> Thanks, >> Sam >> soverdor@REDACTED >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > From lloyd@REDACTED Tue Jun 26 02:06:09 2018 From: lloyd@REDACTED (lloyd@REDACTED) Date: Mon, 25 Jun 2018 20:06:09 -0400 (EDT) Subject: [erlang-questions] String type In-Reply-To: References: Message-ID: <1529971569.247126904@apps.rackspace.com> Hi Richard, I missed this post when it popped up the first time around. But, as usual, it explains much with great clarity. But it still leaves me with profound frustration. At this point I realize that my frustration in part has to do with the highly technical function names in the new and improved string library-- names such as lexemes/2, next_codepoint/1, next_grapheme/1, etc. I do understand that these names are quite precise relative to the concepts they're addressing. But that's the problem for me. When I'm programming in my native language, English, I simply don't think in terms of these concepts. Yes, I could spend a day and build the conceptual bridges between my deprecated ascii-think and the Unicode way of thinking then burn the bridges behind me. But since I don't imagine manipulating Urdu, Chinese, or Swedish text any time soon, as much as I'd love to be fluent in any one of these languages and others, the time spent building and burning bridges feels like an unproductive investment. The technical world is going to Unicode and for good reasons. I get that. But one thing might help clear the fog enormously: A tutorial that explicitly maps the concepts of the deprecated strings to their replacements. The current string reference take a stab. But I find it quite opaque. I know that you're a busy guy. But it seems that you have the skills to clear the fog. Think you squeeze out an hour or two to help an old man and others of my ilk move into the bright and shiny future? All the best, LRP -----Original Message----- From: "Sam Overdorf" Sent: Monday, June 25, 2018 5:12pm To: "Richard O'Keefe" Cc: "Erlang Questions" Subject: Re: [erlang-questions] String type Someone gave me a link to a previous discussion of this and it's problems. I read it and decided that I need to change my process and not modify Erlang. Thanks for the response, Sam On Mon, Jun 25, 2018 at 7:30 AM, Richard O'Keefe wrote: > Yes, people have often considered adding a "real" string > data type to Erlang. With the move to 64-bit machines > this became even more interesting. However, in a Unicode > world, it really is not clear what string *is*. > > For example, in the old ASCII days, it was clear that a > string was a sequence of characters, and all characters > were the same size, and the actual ASCII definition made > it clear that NUL and DEL probably should NOT be allowed > in a string. US programmers (hence US textbooks, hence > practically everyone in the English-speaking world) > quietly ignored the fact that the ASCII standard explicitly > allowed overstrikes so that you could get u-with-umlaut > by doing <"> or even <"> . So in fact in > ASCII a "character" could well be a sequence of code-points > and that is in fact why ` and ^ are in the ASCII set, and > it wasn't therefore *true* that all characters were the > same size. > > In the ISO 8851 family, the standardisers bowed to reality > and forbade overstriking, introducing precomposed accented > letters instead. So the statement that ASCII is a subset > of ISO 8859/1 is a half truth: the codepoints are a subset > but ASCII allows you to DO things with them that Latin-1 > does not. > > Unicode has it both ways. It has precomposed characters > like u-with-umlaut, and it also has composed characters > like u-followed-by-(floating umlaut). Which means we > now have to ask "is a string a sequence of codepoints > or a sequence of characters". But it's more complicated. > See Unicode Technical Annex 29 "Unicode Text Segmentation" > for the horrible details. But the alternatives are > > - sequence of bytes (in UTF8) > - sequence of 16-bit units (UTF16) > - sequence of code-points > + sequence of legacy grapheme clusters > + sequence of extended grapheme clusters > + sequence of tailored grapheme clusters > bearing in mind that > * some code points are always illegal > * most code points are unassigned > * some sequences of code points are illegal > * in particular, legal sequences may have > illegal subsequences, so the "substring" > operation is problematic. > > Let's not even try to think about the existence > of multiple characters with identical appearance, > multiple ways to encode many characters, > invisible characters, characters forbidden by design > then introduced then deprecated, and the question > of whether control marks like redundant direction > indicators should count in deciding whether strings > are equal. > > If you are dealing with text where you are actually > looking at the characters doing some sort of parsing, > the chances are you want a list of tokens or even > some sort of tree rather than a string. > > I'm actually more interested in the fact that you say > you have trouble with lists of strings. Can you > provide an example of the kind of code you have > trouble with? If you use the Dialyzer, it has no > trouble expressing the difference between a list of > integers and a list of lists of integers, and even > without it, it's not a commonly reported problem. > > For example, suppose we have a list of strings and > want to paste them together with spaces between > them. This is called "unwords" in Haskell. Let's > start with the Haskell version. > > unwords :: [String] -> String > unwords [] = [] > unwords (w:ws) = w ++ aux ws > where aux [] = [] > aux (y:ys) = " " ++ y ++ aux ys > > Let's put that into Erlang: > > unwords([]) -> []; > unwords([W|Ws]) -> W ++ unwords_aux(Ws). > > unwords_aux([]) -> ""; > unwords_aux([Y|Ys]) -> " " ++ Y ++ unwords_aux(Ys). > > By the way, this kind of thing is spectacularly > inefficient in languages like Java, which is why Java > has StringBuilder as well as String. This is one of > many reasons why I have a slogan STRINGS ARE WRONG. > > > > On 23 June 2018 at 09:41, Sam Overdorf wrote: >> >> Has anyone considered making string a type and not a list of chars. >> >> I seem to have a lot of trouble when a list is a bunch of string >> objects and I start taking it apart with [H|T] = List.. >> >> When processing the last string in the list I end up taking apart the >> individual characters of the string. If I do a type-check it tells me >> it is a list. >> >> I usually have to do a work around to handle this. If it was a type I >> would easily know when I am done with the list. >> >> Thanks, >> Sam >> soverdor@REDACTED >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From john.hogberg@REDACTED Tue Jun 26 07:50:07 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Tue, 26 Jun 2018 05:50:07 +0000 Subject: [erlang-questions] Patch package OTP 21.0.1 released Message-ID: <1529992207.19459.37.camel@ericsson.com> Patch Package:???????????OTP 21.0.1 Git Tag:?????????????????OTP-21.0.1 Date:????????????????????2018-06-26 Trouble Report Id:???????OTP-15147, OTP-15150 Seq num:?????????????????ERL-644, ERL-650 System:??????????????????OTP Release:?????????????????21 Application:?????????????compiler-7.2.1, erts-10.0.1 Predecessor:?????????????OTP 21.0 ?Check out the git tag OTP-21.0.1, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- compiler-7.2.1 -------------------------------------------------- ?--------------------------------------------------------------------- ?The compiler-7.2.1 application can be applied independently of other ?applications on a full OTP 21 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15150????Application(s): compiler ???????????????Related Id(s): ERL-650 ???????????????The compiler could could crash when compiling a ???????????????complicated function that used the binary syntax. ?Full runtime dependencies of compiler-7.2.1: crypto-3.6, erts-9.0, ?hipe-3.12, kernel-4.0, stdlib-2.5 ?--------------------------------------------------------------------- ?--- erts-10.0.1 ----------------------------------------------------- ?--------------------------------------------------------------------- ?The erts-10.0.1 application can be applied independently of other ?applications on a full OTP 21 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15147????Application(s): erts ???????????????Related Id(s): ERL-644 ???????????????The keys used in os:getenv and os:putenv are ???????????????case-insensitive again on Windows. ?Full runtime dependencies of erts-10.0.1: kernel-6.0, sasl-3.0.1, ?stdlib-3.5 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- From john.hogberg@REDACTED Tue Jun 26 09:26:40 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Tue, 26 Jun 2018 07:26:40 +0000 Subject: [erlang-questions] Patch package OTP 21.0.1 released In-Reply-To: <1529992207.19459.37.camel@ericsson.com> References: <1529992207.19459.37.camel@ericsson.com> Message-ID: <1529997999.19459.42.camel@ericsson.com> The Windows installers have been updated to address the os:getenv/os:putenv case-sensitivity bug (ERL-644). http://www.erlang.org/downloads/21.0 Regards, John H?gberg On tis, 2018-06-26 at 05:50 +0000, John H?gberg wrote: > Patch Package:???????????OTP 21.0.1 > Git Tag:?????????????????OTP-21.0.1 > Date:????????????????????2018-06-26 > Trouble Report Id:???????OTP-15147, OTP-15150 > Seq num:?????????????????ERL-644, ERL-650 > System:??????????????????OTP > Release:?????????????????21 > Application:?????????????compiler-7.2.1, erts-10.0.1 > Predecessor:?????????????OTP 21.0 > > ?Check out the git tag OTP-21.0.1, and build a full OTP system > ?including documentation. Apply one or more applications from this > ?build as patches to your installation using the 'otp_patch_apply' > ?tool. For information on install requirements, see descriptions for > ?each application version below. > > ?------------------------------------------------------------------ > --- > ?--- compiler-7.2.1 ----------------------------------------------- > --- > ?------------------------------------------------------------------ > --- > > ?The compiler-7.2.1 application can be applied independently of other > ?applications on a full OTP 21 installation. > > ?--- Fixed Bugs and Malfunctions --- > > ? OTP-15150????Application(s): compiler > ???????????????Related Id(s): ERL-650 > > ???????????????The compiler could could crash when compiling a > ???????????????complicated function that used the binary syntax. > > > ?Full runtime dependencies of compiler-7.2.1: crypto-3.6, erts-9.0, > ?hipe-3.12, kernel-4.0, stdlib-2.5 > > > ?------------------------------------------------------------------ > --- > ?--- erts-10.0.1 ---------------------------------------------------- > - > ?------------------------------------------------------------------ > --- > > ?The erts-10.0.1 application can be applied independently of other > ?applications on a full OTP 21 installation. > > ?--- Fixed Bugs and Malfunctions --- > > ? OTP-15147????Application(s): erts > ???????????????Related Id(s): ERL-644 > > ???????????????The keys used in os:getenv and os:putenv are > ???????????????case-insensitive again on Windows. > > > ?Full runtime dependencies of erts-10.0.1: kernel-6.0, sasl-3.0.1, > ?stdlib-3.5 > > > ?------------------------------------------------------------------ > --- > ?------------------------------------------------------------------ > --- > ?------------------------------------------------------------------ > --- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Tue Jun 26 11:58:55 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Tue, 26 Jun 2018 18:58:55 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder Message-ID: <1567611.nGgtL7t0W2@takoyaki> With tuple calls gone and mochijson2 finally going extinct I found myself in need of a strings-as-strings JSON encoder/decoder in pure Erlang (not NIF-based). I wrote one yesterday. It is a single module that exports two functions: encode/1 and decode/1. Erlang didn't really *need* another JSON encoder/decoder, of course, but this one works exactly the way I need -- if anyone else finds it useful you're welcome to it: https://gitlab.com/zxq9/zj A few tradeoffs have to be made when writing a JSON encoder/decoder because JSON types don't map very well with Erlang, and everyone has different optimal performance profiles and data idioms within their programs. The mappings ZJ follows are below. Note that tuples map to JSON arrays, which means proplists do NOT map to JSON objects, Erlang maps do. JSON -> Erlang: * Integer -> Integer * Float -> Float * String -> String * Object -> Map * Array -> List * true -> true * false -> false * null -> null Erlang -> JSON: * Integer -> Integer * Float -> Float * Map -> Object * List -> Array * Tuple -> Array * Binary -> String * UTF-8 -> String * true -> true * false -> false * null -> null From essen@REDACTED Tue Jun 26 12:43:43 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 26 Jun 2018 12:43:43 +0200 Subject: [erlang-questions] Patch package OTP 21.0.1 released In-Reply-To: <1529992207.19459.37.camel@ericsson.com> References: <1529992207.19459.37.camel@ericsson.com> Message-ID: <189828ed-0ca0-a81a-e5aa-c551ba414c40@ninenines.eu> Hello, Do you by any chance have the changelog for all patch releases conveniently available somewhere? I am thinking of storing them all together somewhere to make them easy to search through when trying to debug issues between versions. If they are all on hand but private please send me a tarball for 18+. Thanks! On 06/26/2018 07:50 AM, John H?gberg wrote: > Patch Package:???????????OTP 21.0.1 > Git Tag:?????????????????OTP-21.0.1 > Date:????????????????????2018-06-26 > Trouble Report Id:???????OTP-15147, OTP-15150 > Seq num:?????????????????ERL-644, ERL-650 > System:??????????????????OTP > Release:?????????????????21 > Application:?????????????compiler-7.2.1, erts-10.0.1 > Predecessor:?????????????OTP 21.0 > > ?Check out the git tag OTP-21.0.1, and build a full OTP system > ?including documentation. Apply one or more applications from this > ?build as patches to your installation using the 'otp_patch_apply' > ?tool. For information on install requirements, see descriptions for > ?each application version below. > > ?--------------------------------------------------------------------- > ?--- compiler-7.2.1 -------------------------------------------------- > ?--------------------------------------------------------------------- > > ?The compiler-7.2.1 application can be applied independently of other > ?applications on a full OTP 21 installation. > > ?--- Fixed Bugs and Malfunctions --- > > ? OTP-15150????Application(s): compiler > ???????????????Related Id(s): ERL-650 > > ???????????????The compiler could could crash when compiling a > ???????????????complicated function that used the binary syntax. > > > ?Full runtime dependencies of compiler-7.2.1: crypto-3.6, erts-9.0, > ?hipe-3.12, kernel-4.0, stdlib-2.5 > > > ?--------------------------------------------------------------------- > ?--- erts-10.0.1 ----------------------------------------------------- > ?--------------------------------------------------------------------- > > ?The erts-10.0.1 application can be applied independently of other > ?applications on a full OTP 21 installation. > > ?--- Fixed Bugs and Malfunctions --- > > ? OTP-15147????Application(s): erts > ???????????????Related Id(s): ERL-644 > > ???????????????The keys used in os:getenv and os:putenv are > ???????????????case-insensitive again on Windows. > > > ?Full runtime dependencies of erts-10.0.1: kernel-6.0, sasl-3.0.1, > ?stdlib-3.5 > > > ?--------------------------------------------------------------------- > ?--------------------------------------------------------------------- > ?--------------------------------------------------------------------- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From john.hogberg@REDACTED Tue Jun 26 12:50:50 2018 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Tue, 26 Jun 2018 10:50:50 +0000 Subject: [erlang-questions] Patch package OTP 21.0.1 released In-Reply-To: <189828ed-0ca0-a81a-e5aa-c551ba414c40@ninenines.eu> References: <1529992207.19459.37.camel@ericsson.com> <189828ed-0ca0-a81a-e5aa-c551ba414c40@ninenines.eu> Message-ID: <1530010250.19459.53.camel@ericsson.com> The README's at http://erlang.org/download/?(without the `s`!) look complete from OTP 17 onwards. /John On tis, 2018-06-26 at 12:43 +0200, Lo?c Hoguin wrote: > Hello, > > Do you by any chance have the changelog for all patch releases? > conveniently available somewhere? I am thinking of storing them all? > together somewhere to make them easy to search through when trying > to? > debug issues between versions. If they are all on hand but private? > please send me a tarball for 18+. > > Thanks! > > On 06/26/2018 07:50 AM, John H?gberg wrote: > > > > Patch Package:???????????OTP 21.0.1 > > Git Tag:?????????????????OTP-21.0.1 > > Date:????????????????????2018-06-26 > > Trouble Report Id:???????OTP-15147, OTP-15150 > > Seq num:?????????????????ERL-644, ERL-650 > > System:??????????????????OTP > > Release:?????????????????21 > > Application:?????????????compiler-7.2.1, erts-10.0.1 > > Predecessor:?????????????OTP 21.0 > > > > ??Check out the git tag OTP-21.0.1, and build a full OTP system > > ??including documentation. Apply one or more applications from this > > ??build as patches to your installation using the 'otp_patch_apply' > > ??tool. For information on install requirements, see descriptions > > for > > ??each application version below. > > > > ??----------------------------------------------------------------- > > ---- > > ??--- compiler-7.2.1 ---------------------------------------------- > > ---- > > ??----------------------------------------------------------------- > > ---- > > > > ??The compiler-7.2.1 application can be applied independently of > > other > > ??applications on a full OTP 21 installation. > > > > ??--- Fixed Bugs and Malfunctions --- > > > > ?? OTP-15150????Application(s): compiler > > ????????????????Related Id(s): ERL-650 > > > > ????????????????The compiler could could crash when compiling a > > ????????????????complicated function that used the binary syntax. > > > > > > ??Full runtime dependencies of compiler-7.2.1: crypto-3.6, erts- > > 9.0, > > ??hipe-3.12, kernel-4.0, stdlib-2.5 > > > > > > ??----------------------------------------------------------------- > > ---- > > ??--- erts-10.0.1 ----------------------------------------------- > > ------ > > ??----------------------------------------------------------------- > > ---- > > > > ??The erts-10.0.1 application can be applied independently of other > > ??applications on a full OTP 21 installation. > > > > ??--- Fixed Bugs and Malfunctions --- > > > > ?? OTP-15147????Application(s): erts > > ????????????????Related Id(s): ERL-644 > > > > ????????????????The keys used in os:getenv and os:putenv are > > ????????????????case-insensitive again on Windows. > > > > > > ??Full runtime dependencies of erts-10.0.1: kernel-6.0, sasl-3.0.1, > > ??stdlib-3.5 > > > > > > ??----------------------------------------------------------------- > > ---- > > ??----------------------------------------------------------------- > > ---- > > ??----------------------------------------------------------------- > > ---- > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > From essen@REDACTED Tue Jun 26 12:54:22 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 26 Jun 2018 12:54:22 +0200 Subject: [erlang-questions] Patch package OTP 21.0.1 released In-Reply-To: <1530010250.19459.53.camel@ericsson.com> References: <1529992207.19459.37.camel@ericsson.com> <189828ed-0ca0-a81a-e5aa-c551ba414c40@ninenines.eu> <1530010250.19459.53.camel@ericsson.com> Message-ID: <32014e50-468c-38dc-9368-691bf0523411@ninenines.eu> Gold mine! Cheers. On 06/26/2018 12:50 PM, John H?gberg wrote: > The README's at http://erlang.org/download/?(without the `s`!) look > complete from OTP 17 onwards. > > /John > > On tis, 2018-06-26 at 12:43 +0200, Lo?c Hoguin wrote: >> Hello, >> >> Do you by any chance have the changelog for all patch releases >> conveniently available somewhere? I am thinking of storing them all >> together somewhere to make them easy to search through when trying >> to >> debug issues between versions. If they are all on hand but private >> please send me a tarball for 18+. >> >> Thanks! >> >> On 06/26/2018 07:50 AM, John H?gberg wrote: >>> >>> Patch Package:???????????OTP 21.0.1 >>> Git Tag:?????????????????OTP-21.0.1 >>> Date:????????????????????2018-06-26 >>> Trouble Report Id:???????OTP-15147, OTP-15150 >>> Seq num:?????????????????ERL-644, ERL-650 >>> System:??????????????????OTP >>> Release:?????????????????21 >>> Application:?????????????compiler-7.2.1, erts-10.0.1 >>> Predecessor:?????????????OTP 21.0 >>> >>> ??Check out the git tag OTP-21.0.1, and build a full OTP system >>> ??including documentation. Apply one or more applications from this >>> ??build as patches to your installation using the 'otp_patch_apply' >>> ??tool. For information on install requirements, see descriptions >>> for >>> ??each application version below. >>> >>> ??----------------------------------------------------------------- >>> ---- >>> ??--- compiler-7.2.1 ---------------------------------------------- >>> ---- >>> ??----------------------------------------------------------------- >>> ---- >>> >>> ??The compiler-7.2.1 application can be applied independently of >>> other >>> ??applications on a full OTP 21 installation. >>> >>> ??--- Fixed Bugs and Malfunctions --- >>> >>> ?? OTP-15150????Application(s): compiler >>> ????????????????Related Id(s): ERL-650 >>> >>> ????????????????The compiler could could crash when compiling a >>> ????????????????complicated function that used the binary syntax. >>> >>> >>> ??Full runtime dependencies of compiler-7.2.1: crypto-3.6, erts- >>> 9.0, >>> ??hipe-3.12, kernel-4.0, stdlib-2.5 >>> >>> >>> ??----------------------------------------------------------------- >>> ---- >>> ??--- erts-10.0.1 ----------------------------------------------- >>> ------ >>> ??----------------------------------------------------------------- >>> ---- >>> >>> ??The erts-10.0.1 application can be applied independently of other >>> ??applications on a full OTP 21 installation. >>> >>> ??--- Fixed Bugs and Malfunctions --- >>> >>> ?? OTP-15147????Application(s): erts >>> ????????????????Related Id(s): ERL-644 >>> >>> ????????????????The keys used in os:getenv and os:putenv are >>> ????????????????case-insensitive again on Windows. >>> >>> >>> ??Full runtime dependencies of erts-10.0.1: kernel-6.0, sasl-3.0.1, >>> ??stdlib-3.5 >>> >>> >>> ??----------------------------------------------------------------- >>> ---- >>> ??----------------------------------------------------------------- >>> ---- >>> ??----------------------------------------------------------------- >>> ---- >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions -- Lo?c Hoguin https://ninenines.eu From henrik.x.nord@REDACTED Tue Jun 26 13:06:38 2018 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Tue, 26 Jun 2018 11:06:38 +0000 Subject: [erlang-questions] Patch package OTP 21.0.1 released In-Reply-To: <1530010250.19459.53.camel@ericsson.com> References: <1529992207.19459.37.camel@ericsson.com> <189828ed-0ca0-a81a-e5aa-c551ba414c40@ninenines.eu> <1530010250.19459.53.camel@ericsson.com> Message-ID: <1530011197.5596.26.camel@ericsson.com> On tis, 2018-06-26 at 10:50 +0000, John H?gberg wrote: > The README's at http://erlang.org/download/?(without the `s`!) look > complete from OTP 17 onwards. > > /John From R7B-1 even :D otp_src_R7B-1.readme? some are in the old versions as above, and from?OTP-17.0.1.README all small patches should have readme?s in the new format. you can also find this information in the documentation under release notes. http://erlang.org/doc/apps/erts/notes.html#erts-10.0 Note that we do not update the online documentation until we release a planned patch package. But you should be able to see all the 9.XXX versions of erts in there. But you can always build the doc yourself. /Henrik > > On tis, 2018-06-26 at 12:43 +0200, Lo?c Hoguin wrote: > > > > Hello, > > > > Do you by any chance have the changelog for all patch releases? > > conveniently available somewhere? I am thinking of storing them > > all? > > together somewhere to make them easy to search through when trying > > to? > > debug issues between versions. If they are all on hand but private? > > please send me a tarball for 18+. > > > > Thanks! > > > > On 06/26/2018 07:50 AM, John H?gberg wrote: > > > > > > > > > Patch Package:???????????OTP 21.0.1 > > > Git Tag:?????????????????OTP-21.0.1 > > > Date:????????????????????2018-06-26 > > > Trouble Report Id:???????OTP-15147, OTP-15150 > > > Seq num:?????????????????ERL-644, ERL-650 > > > System:??????????????????OTP > > > Release:?????????????????21 > > > Application:?????????????compiler-7.2.1, erts-10.0.1 > > > Predecessor:?????????????OTP 21.0 > > > > > > ??Check out the git tag OTP-21.0.1, and build a full OTP system > > > ??including documentation. Apply one or more applications from > > > this > > > ??build as patches to your installation using the > > > 'otp_patch_apply' > > > ??tool. For information on install requirements, see descriptions > > > for > > > ??each application version below. > > > > > > ??--------------------------------------------------------------- > > > -- > > > ---- > > > ??--- compiler-7.2.1 -------------------------------------------- > > > -- > > > ---- > > > ??--------------------------------------------------------------- > > > -- > > > ---- > > > > > > ??The compiler-7.2.1 application can be applied independently of > > > other > > > ??applications on a full OTP 21 installation. > > > > > > ??--- Fixed Bugs and Malfunctions --- > > > > > > ?? OTP-15150????Application(s): compiler > > > ????????????????Related Id(s): ERL-650 > > > > > > ????????????????The compiler could could crash when compiling a > > > ????????????????complicated function that used the binary syntax. > > > > > > > > > ??Full runtime dependencies of compiler-7.2.1: crypto-3.6, erts- > > > 9.0, > > > ??hipe-3.12, kernel-4.0, stdlib-2.5 > > > > > > > > > ??--------------------------------------------------------------- > > > -- > > > ---- > > > ??--- erts-10.0.1 ----------------------------------------------- > > > ------ > > > ??--------------------------------------------------------------- > > > -- > > > ---- > > > > > > ??The erts-10.0.1 application can be applied independently of > > > other > > > ??applications on a full OTP 21 installation. > > > > > > ??--- Fixed Bugs and Malfunctions --- > > > > > > ?? OTP-15147????Application(s): erts > > > ????????????????Related Id(s): ERL-644 > > > > > > ????????????????The keys used in os:getenv and os:putenv are > > > ????????????????case-insensitive again on Windows. > > > > > > > > > ??Full runtime dependencies of erts-10.0.1: kernel-6.0, sasl- > > > 3.0.1, > > > ??stdlib-3.5 > > > > > > > > > ??--------------------------------------------------------------- > > > -- > > > ---- > > > ??--------------------------------------------------------------- > > > -- > > > ---- > > > ??--------------------------------------------------------------- > > > -- > > > ---- > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Tue Jun 26 14:18:56 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 26 Jun 2018 14:18:56 +0200 Subject: [erlang-questions] Patch package OTP 21.0.1 released In-Reply-To: <1530011197.5596.26.camel@ericsson.com> References: <1529992207.19459.37.camel@ericsson.com> <189828ed-0ca0-a81a-e5aa-c551ba414c40@ninenines.eu> <1530010250.19459.53.camel@ericsson.com> <1530011197.5596.26.camel@ericsson.com> Message-ID: <562fcc6c-5dcb-2ada-5705-18518b860bad@ninenines.eu> On 06/26/2018 01:06 PM, Henrik Nord X wrote: > On tis, 2018-06-26 at 10:50 +0000, John H?gberg wrote: >> The README's at http://erlang.org/download/?(without the `s`!) look >> complete from OTP 17 onwards. >> >> /John > From R7B-1 even :D > otp_src_R7B-1.readme > some are in the old versions as above, and from?OTP-17.0.1.README all > small patches should have readme?s in the new format. > you can also find this information in the documentation under release > notes. > http://erlang.org/doc/apps/erts/notes.html#erts-10.0 > > Note that we do not update the online documentation until we release a > planned patch package. But you should be able to see all the 9.XXX > versions of erts in there. > > But you can always build the doc yourself. I noted that the following versions were announced but do not seem to have a README so I got the README from the announcement itself: * 18.3.4.4 * 18.3.4.5 * 18.3.4.6 * 18.3.4.8 * 19.0.4 * 19.1.2 * 19.3.4 * 19.3.6.2 * 19.3.6.7 * 20.1.3 Maybe you're interested in updating the website with them. I can't find an announcement or a README for the following two tags: * 18.3.4.3 * 19.2.3.1 I don't expect a README or an announcement for 19.2.3.1 but I'm a bit more surprised about 18.3.4.3, maybe there's a good reason? Cheers, -- Lo?c Hoguin https://ninenines.eu From essen@REDACTED Tue Jun 26 14:28:14 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 26 Jun 2018 14:28:14 +0200 Subject: [erlang-questions] SSL doc issues noticed Message-ID: <2d9b9cf6-59a7-eb11-6245-965bfb81abea@ninenines.eu> Hello, bugs.erlang.org is down. I noticed the following two issues in the SSL docs: * https://github.com/erlang/otp/blob/maint/lib/ssl/doc/src/ssl.xml#L202 * https://github.com/erlang/otp/blob/maint/lib/ssl/doc/src/ssl.xml#L207 Seems like something is missing but I do not know what. Cheers, -- Lo?c Hoguin https://ninenines.eu From henrik.x.nord@REDACTED Tue Jun 26 15:10:24 2018 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Tue, 26 Jun 2018 13:10:24 +0000 Subject: [erlang-questions] Patch package OTP 21.0.1 released In-Reply-To: <562fcc6c-5dcb-2ada-5705-18518b860bad@ninenines.eu> References: <1529992207.19459.37.camel@ericsson.com> <189828ed-0ca0-a81a-e5aa-c551ba414c40@ninenines.eu> <1530010250.19459.53.camel@ericsson.com> <1530011197.5596.26.camel@ericsson.com> <562fcc6c-5dcb-2ada-5705-18518b860bad@ninenines.eu> Message-ID: <1530018624.5596.33.camel@ericsson.com> On tis, 2018-06-26 at 14:18 +0200, Lo?c Hoguin wrote: > On 06/26/2018 01:06 PM, Henrik Nord X wrote: > > > > On tis, 2018-06-26 at 10:50 +0000, John H?gberg wrote: > > > > > > The README's at http://erlang.org/download/?(without the `s`!) > > > look > > > complete from OTP 17 onwards. > > > > > > /John > > ?From R7B-1 even :D > > otp_src_R7B-1.readme > > some are in the old versions as above, and from?OTP-17.0.1.README > > all > > small patches should have readme?s in the new format. > > you can also find this information in the documentation under > > release > > notes. > > http://erlang.org/doc/apps/erts/notes.html#erts-10.0 > > > > Note that we do not update the online documentation until we > > release a > > planned patch package. But you should be able to see all the 9.XXX > > versions of erts in there. > > > > But you can always build the doc yourself. > I noted that the following versions were announced but do not seem > to? > have a README so I got the README from the announcement itself: > > * 18.3.4.4 > * 18.3.4.5 > * 18.3.4.6 > * 18.3.4.8 > * 19.0.4 > * 19.1.2 > * 19.3.4 > * 19.3.6.2 > * 19.3.6.7 > * 20.1.3 > > Maybe you're interested in updating the website with them. Done > > I can't find an announcement or a README for the following two tags: > > * 18.3.4.3 > * 19.2.3.1 > I do not know why there are no announcement on some of the patches, but the patches are still published and thus we have a README, now also on the download page > I don't expect a README or an announcement for 19.2.3.1 but I'm a > bit? > more surprised about 18.3.4.3, maybe there's a good reason? > There might have been some kind of decision to not post readme for some of the patches that are already fixed upstream, when backported to an older OTP version. As the changenote for that should already be posted. Keep in mind the versioning and what that implies to what code is included in what following version etc. I.E the 19.2.3.1 patch is a backport for something that is released in 21.0 > Cheers, > From essen@REDACTED Tue Jun 26 15:21:07 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 26 Jun 2018 15:21:07 +0200 Subject: [erlang-questions] Patch package OTP 21.0.1 released In-Reply-To: <1530018624.5596.33.camel@ericsson.com> References: <1529992207.19459.37.camel@ericsson.com> <189828ed-0ca0-a81a-e5aa-c551ba414c40@ninenines.eu> <1530010250.19459.53.camel@ericsson.com> <1530011197.5596.26.camel@ericsson.com> <562fcc6c-5dcb-2ada-5705-18518b860bad@ninenines.eu> <1530018624.5596.33.camel@ericsson.com> Message-ID: <4becaf5d-b695-c2b4-2abe-8d1002842278@ninenines.eu> On 06/26/2018 03:10 PM, Henrik Nord X wrote:>> Maybe you're interested in updating the website with them. > Done \o/ >> I can't find an announcement or a README for the following two tags: >> >> * 18.3.4.3 >> * 19.2.3.1 >> > I do not know why there are no announcement on some of the patches, but > the patches are still published and thus we have a README, now also on > the download page \o/ >> I don't expect a README or an announcement for 19.2.3.1 but I'm a >> bit >> more surprised about 18.3.4.3, maybe there's a good reason? >> > There might have been some kind of decision to not post readme for some > of the patches that are already fixed upstream, when backported to an > older OTP version. As the changenote for that should already be posted. > > Keep in mind the versioning and what that implies to what code is > included in what following version etc. > > I.E the 19.2.3.1 patch is a backport for something that is released in > 21.0 No worries there, I personally always test with the last announced versions for each minor release. But users sometimes get creative and this helps figuring things out. For anyone interested I have added all READMEs to https://git.ninenines.eu/ci.erlang.mk.git/tree/release-notes and will continue adding them as they get released. Older versions might be removed in time though. Cheers, -- Lo?c Hoguin https://ninenines.eu From dmkolesnikov@REDACTED Tue Jun 26 15:44:08 2018 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 26 Jun 2018 16:44:08 +0300 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <1567611.nGgtL7t0W2@takoyaki> References: <1567611.nGgtL7t0W2@takoyaki> Message-ID: Hello, It looks promising, thank you for sharing! However, I?d like to suggest to consider mapping `null -> null`. I?ve been raising a similar issue to jsx as well. The atom `undefined` is used by many library to express undefined result. Please consider its usage JSON -> Erlang: * null -> undefined Erlang -> JSON * undefined -> null Best Regards, Dmitry > On 26 Jun 2018, at 12.58, zxq9@REDACTED wrote: > > With tuple calls gone and mochijson2 finally going extinct I found myself in need of a strings-as-strings JSON encoder/decoder in pure Erlang (not NIF-based). I wrote one yesterday. It is a single module that exports two functions: encode/1 and decode/1. Erlang didn't really *need* another JSON encoder/decoder, of course, but this one works exactly the way I need -- if anyone else finds it useful you're welcome to it: > > https://gitlab.com/zxq9/zj > > A few tradeoffs have to be made when writing a JSON encoder/decoder because JSON types don't map very well with Erlang, and everyone has different optimal performance profiles and data idioms within their programs. The mappings ZJ follows are below. Note that tuples map to JSON arrays, which means proplists do NOT map to JSON objects, Erlang maps do. > > JSON -> Erlang: > * Integer -> Integer > * Float -> Float > * String -> String > * Object -> Map > * Array -> List > * true -> true > * false -> false > * null -> null > > Erlang -> JSON: > * Integer -> Integer > * Float -> Float > * Map -> Object > * List -> Array > * Tuple -> Array > * Binary -> String > * UTF-8 -> String > * true -> true > * false -> false > * null -> null > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From christopher.casilli@REDACTED Tue Jun 26 19:43:19 2018 From: christopher.casilli@REDACTED (chris casilli) Date: Tue, 26 Jun 2018 13:43:19 -0400 Subject: [erlang-questions] Different result using io_lib:format under version 21, Is this expected? Message-ID: <105C58C6-7E7E-4835-AD9D-3580B5E72F8D@gmail.com> Hello, A few of my tests started failing when run under the newer versions 21.0. I wanted to ensure this is expected new behavior, before I modify my code to match new results: With the previous versions this was the result: io_lib:format(?~.16B?,[255]). [?FF?] This is the current result under the new version 21.0 & 20.0.1 io_lib:format(?~.16B?,[255]). ?FF? Please let me know if this is expected and the new expected result. Kind Regards, Chris Casilli From solvip@REDACTED Tue Jun 26 22:04:12 2018 From: solvip@REDACTED (=?UTF-8?B?U8O2bHZpIFDDoWxsIMOBc2dlaXJzc29u?=) Date: Tue, 26 Jun 2018 20:04:12 +0000 Subject: [erlang-questions] Different result using io_lib:format under version 21, Is this expected? In-Reply-To: <105C58C6-7E7E-4835-AD9D-3580B5E72F8D@gmail.com> References: <105C58C6-7E7E-4835-AD9D-3580B5E72F8D@gmail.com> Message-ID: Hi Chris Both are legal return values from io_lib:format/2; it's defined such that it might return deep lists. I'd suggest that you flatten the output in your tests and don't rely on a specific structure. You can do that by using e.g. the built-in function iolist_to_binary/1 or lists:flatten/1. However, you do not need to, and should not, flatten the list before writing it to a file or a socket. I've found that https://prog21.dadgum.com/70.html explains the concept quite well. Regards S?lvi P?ll ?. On Tue, Jun 26, 2018 at 5:43 PM, chris casilli wrote: > Hello, > > A few of my tests started failing when run under the newer versions 21.0. > I wanted to ensure this is expected new behavior, before I modify my code to match new results: > > With the previous versions this was the result: > io_lib:format(?~.16B?,[255]). > [?FF?] > > This is the current result under the new version 21.0 & 20.0.1 > io_lib:format(?~.16B?,[255]). > ?FF? > > Please let me know if this is expected and the new expected result. > > Kind Regards, > > Chris Casilli > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Wed Jun 27 06:29:34 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 27 Jun 2018 13:29:34 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <1567611.nGgtL7t0W2@takoyaki> References: <1567611.nGgtL7t0W2@takoyaki> Message-ID: <1752242.xhRncR51Ln@takoyaki> On 2018?6?26???? 18?58?55? JST zxq9@REDACTED wrote: > With tuple calls gone and mochijson2 finally going extinct I found > myself in need of a strings-as-strings JSON encoder/decoder in pure > Erlang (not NIF-based). I wrote one yesterday. It is a single module > that exports two functions: encode/1 and decode/1. Erlang didn't really > *need* another JSON encoder/decoder, of course, but this one works > exactly the way I need -- if anyone else finds it useful you're welcome > to it: > > https://gitlab.com/zxq9/zj Quick update... Two new functions have been added, binary_encode/1 and binary_decode/1. The purpose of these new functions is to disambiguate between strings and arrays. Most of the time this is not necessary (it seems the vast majority of JSON data floating around is snippets of ASCII), but it is easy to provide for so I added these this morning. Since someone is already using this I'm calling it v1.0.0, as I intend to support this 4-function interface for a while. https://gitlab.com/zxq9/zj#binary_decode1 Just one more option in the ocean of JSON encode/decode libs! -Craig From zxq9@REDACTED Wed Jun 27 06:34:42 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 27 Jun 2018 13:34:42 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> Message-ID: <28870908.8m8mkDmdhQ@takoyaki> On 2018?6?26???? 16?44?08? JST you wrote: > Hello, > > It looks promising, thank you for sharing! > > However, I?d like to suggest to consider mapping `null -> null`. > I?ve been raising a similar issue to jsx as well. > > The atom `undefined` is used by many library to express undefined result. > Please consider its usage > > JSON -> Erlang: > * null -> undefined > > Erlang -> JSON > * undefined -> null Hmmm... That changes semantics a bit. I understand what you're saying, but that would mean: Erlang -> JSON - null -> null - undefined -> null Is that really desirable? Seems a bit "fuzzy"/surprising. I'm not faced with this anywhere, so I really don't know. Thoughts from anyone else? -Craig From max.lapshin@REDACTED Wed Jun 27 08:06:22 2018 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 27 Jun 2018 09:06:22 +0300 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <28870908.8m8mkDmdhQ@takoyaki> References: <1567611.nGgtL7t0W2@takoyaki> <28870908.8m8mkDmdhQ@takoyaki> Message-ID: I also think that erlang undefined should be converted to json null and back. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed Jun 27 09:05:59 2018 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 27 Jun 2018 08:05:59 +0100 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> <28870908.8m8mkDmdhQ@takoyaki> Message-ID: If we were dealing with records, I'd argue that 'undefined' should map to *omitted*, but we're dealing with maps, so... (shrugs). Incidentally: - https://github.com/davisp/jiffy/issues/4 - https://github.com/davisp/jiffy/issues/27 - https://github.com/talentdeficit/jsx/issues/18 On 27 June 2018 at 07:06, Max Lapshin wrote: > I also think that erlang undefined should be converted to json null and > back. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From zxq9@REDACTED Wed Jun 27 09:21:53 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 27 Jun 2018 16:21:53 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> Message-ID: <2063693.FJ7v3LWtuf@takoyaki> On 2018?6?27???? 8?05?59? JST Roger Lipscombe wrote: > If we were dealing with records, I'd argue that 'undefined' should map > to *omitted*, but we're dealing with maps, so... (shrugs). > > Incidentally: > - https://github.com/davisp/jiffy/issues/4 > - https://github.com/davisp/jiffy/issues/27 > - https://github.com/talentdeficit/jsx/issues/18 > > On 27 June 2018 at 07:06, Max Lapshin wrote: > > I also think that erlang undefined should be converted to json null and > > back. Indeed. On reflection the argument for translating the semantic VS merely passing along the label that conveys it has won me over. I pulled the v1.0.0 tag that I had *just* pushed and reworked it. ZJ's type mapping: https://gitlab.com/zxq9/zj#type-mapping Erlang -> JSON - true -> true - false -> false - undefined -> null - Atom -> String JSON -> Erlang - true -> true - false -> false - null -> undefined Note that non-special atoms map to strings, and that includes 'null' now mapping to "null". Thanks, fellas. -Craig From michael.nisi@REDACTED Wed Jun 27 09:42:14 2018 From: michael.nisi@REDACTED (Michael Nisi) Date: Wed, 27 Jun 2018 09:42:14 +0200 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <2063693.FJ7v3LWtuf@takoyaki> References: <1567611.nGgtL7t0W2@takoyaki> <2063693.FJ7v3LWtuf@takoyaki> Message-ID: <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> Here?s how v8::JSON, the JSON parser in Node?s JavaScript engine, does it: > JSON.stringify({}) '{}' > JSON.stringify({ name: null }) '{"name":null}' > JSON.stringify({ name: undefined }) '{}' > JSON.stringify({ name: 'Lionel' }) '{"name":"Lionel"}? > JSON.parse('{}').name undefined > JSON.parse('{ "name": null }').name null JavaScript differentiates between null and undefined, without wanting to get philosophical here. Michael > On 27. Jun 2018, at 09:21, zxq9@REDACTED wrote: > > Erlang -> JSON > - true -> true > - false -> false > - undefined -> null > - Atom -> String > > JSON -> Erlang > - true -> true > - false -> false > - null -> undefined > From marc@REDACTED Wed Jun 27 10:05:04 2018 From: marc@REDACTED (Marc Worrell) Date: Wed, 27 Jun 2018 10:05:04 +0200 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> References: <1567611.nGgtL7t0W2@takoyaki> <2063693.FJ7v3LWtuf@takoyaki> <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> Message-ID: <7BC82C2F-00B0-47A4-82ED-142E5695CB98@worrell.nl> I agree with mapping ?null' to ?undefined' and vice versa. An issue with ?null? in JavaScript versus Erlang is that in JavaScript ?null? does have meaning. In Erlang it is just another random atom. Consider in JavaScript: > if (null) console.log('yeah'); < undefined > if (!null) console.log('yeah'); [Log] yeah In Erlang libraries only ?undefined? is handled. If you need to interface the received JSON to Erlang then you need to either change the libraries or add mapping code. To me it seems to be pragmatic to let the decoder use ?undefined? instead of ?null?. - Marc > On 27 Jun 2018, at 09:42, Michael Nisi wrote: > > Here?s how v8::JSON, the JSON parser in Node?s JavaScript engine, does it: > >> JSON.stringify({}) > '{}' >> JSON.stringify({ name: null }) > '{"name":null}' >> JSON.stringify({ name: undefined }) > '{}' >> JSON.stringify({ name: 'Lionel' }) > '{"name":"Lionel"}? > >> JSON.parse('{}').name > undefined >> JSON.parse('{ "name": null }').name > null > > JavaScript differentiates between null and undefined, without wanting to get philosophical here. > > Michael > > >> On 27. Jun 2018, at 09:21, zxq9@REDACTED wrote: >> >> Erlang -> JSON >> - true -> true >> - false -> false >> - undefined -> null >> - Atom -> String >> >> JSON -> Erlang >> - true -> true >> - false -> false >> - null -> undefined >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Wed Jun 27 10:37:23 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 27 Jun 2018 17:37:23 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <1752242.xhRncR51Ln@takoyaki> References: <1567611.nGgtL7t0W2@takoyaki> <1752242.xhRncR51Ln@takoyaki> Message-ID: <1935958.IorJDrbWiJ@takoyaki> On 2018?6?27???? 13?29?34? JST zxq9@REDACTED wrote: > On 2018?6?26???? 18?58?55? JST zxq9@REDACTED wrote: > > With tuple calls gone and mochijson2 finally going extinct I found > > myself in need of a strings-as-strings JSON encoder/decoder in pure > > Erlang (not NIF-based). I wrote one yesterday. It is a single module > > that exports two functions: encode/1 and decode/1. Erlang didn't really > > *need* another JSON encoder/decoder, of course, but this one works > > exactly the way I need -- if anyone else finds it useful you're welcome > > to it: > > > > https://gitlab.com/zxq9/zj > > > Quick update... > > Two new functions have been added, binary_encode/1 and binary_decode/1. > The purpose of these new functions is to disambiguate between strings > and arrays. Most of the time this is not necessary (it seems the vast > majority of JSON data floating around is snippets of ASCII), but it is > easy to provide for so I added these this morning. > > Since someone is already using this I'm calling it v1.0.0, as I intend > to support this 4-function interface for a while. > > https://gitlab.com/zxq9/zj#binary_decode1 > > Just one more option in the ocean of JSON encode/decode libs! And one final status update... Marc Worrell kindly reminded me that I still had GPL-3.0 as the license. That's changed. ZJ is now under the MIT license. -Craig From chris.whealy@REDACTED Wed Jun 27 10:43:39 2018 From: chris.whealy@REDACTED (Whealy, Chris) Date: Wed, 27 Jun 2018 08:43:39 +0000 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> References: <1567611.nGgtL7t0W2@takoyaki> <2063693.FJ7v3LWtuf@takoyaki> <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> Message-ID: Allow me to elaborate on your point Michael (also without getting philosophical). In JavaScript null and undefined are identifiably distinct datatypes that serve specific purposes: * null - This variable specifically has no value * undefined - The value of this variable is indeterminate (I.E. identifiably different from null) Although the atom null has no special meaning in Erlang, it does when mapped to JavaScript null; therefore to maintain accuracy, Erlang devs who also work in JavaScript should understand and preserve this semantic difference. Likewise with Erlang undefined mapping to JavaScript undefined. Therefore, I submit that this semantic difference should be persevered when mapping from Erlang to JavaScript, otherwise data loss will occur, particularly when mapping from JavaScript back to Erlang. Erlang -> JavaScript null -> null undefined -> undefined JavaScript -> Erlang null -> null undefined -> undefined This part of the mapping table at least should be bijective. Chris Whealy SAP Cloud Platform | Strategy & Product Management | Team SAP UK Ltd, Clockhouse Place, Bedfont Rd, Feltham, Middx, TW14 8HA, England M +44 (0)7808 575377 Find out more on the Strategy & Product Management Wiki page (SAP internal) Follow our latest activities in SAP CP User Community Jam Group Please consider the impact on the environment before printing this e-mail. Twitter: @LogaRhythm "The voice of ignorance speaks loud and long, but the words of the wise are quiet and few" Ancient Proverb On 27 Jun 2018, at 08:42, Michael Nisi > wrote: Here?s how v8::JSON, the JSON parser in Node?s JavaScript engine, does it: JSON.stringify({}) '{}' JSON.stringify({ name: null }) '{"name":null}' JSON.stringify({ name: undefined }) '{}' JSON.stringify({ name: 'Lionel' }) '{"name":"Lionel"}? JSON.parse('{}').name undefined JSON.parse('{ "name": null }').name null JavaScript differentiates between null and undefined, without wanting to get philosophical here. Michael On 27. Jun 2018, at 09:21, zxq9@REDACTED wrote: Erlang -> JSON - true -> true - false -> false - undefined -> null - Atom -> String JSON -> Erlang - true -> true - false -> false - null -> undefined _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Jun 27 10:53:00 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 27 Jun 2018 17:53:00 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> Message-ID: <1820955.6qJRxnCLPm@takoyaki> On 2018?6?27???? 8?43?39? JST you wrote: > Allow me to elaborate on your point Michael (also without getting philosophical). > > In JavaScript null and undefined are identifiably distinct datatypes that serve specific purposes: > > > * null - This variable specifically has no value > * undefined - The value of this variable is indeterminate (I.E. identifiably different from null) > > Although the atom null has no special meaning in Erlang, it does when mapped to JavaScript null; therefore to maintain accuracy, Erlang devs who also work in JavaScript should understand and preserve this semantic difference. Likewise with Erlang undefined mapping to JavaScript undefined. > > Therefore, I submit that this semantic difference should be persevered when mapping from Erlang to JavaScript, otherwise data loss will occur, particularly when mapping from JavaScript back to Erlang. > > Erlang -> JavaScript > null -> null > undefined -> undefined > > JavaScript -> Erlang > null -> null > undefined -> undefined > > This part of the mapping table at least should be bijective. If I were mapping JavaScript I would agree -- but I am not, I am mapping JSON as defined by RFC-8259, which is distinct from JavaScript/ECMAScript. https://tools.ietf.org/html/rfc8259 RFC-8259 does not define the atom 'undefined' at all, whereas Erlang does. Most existing JSON encoders/decoders either don't map "undefined" (since it shouldn't occur in JSON at all) or map "undefined" -> 'undefined' as you suggest above. ZJ provides a (clearly desired) alternative to this. Fortunately there are enough JSON encoders/decoders around that finding the flavor that fits a given project best is mostly a matter of shopping around and testing. That said, there is a little wiggle room for real-world flexibility. ZJ accepts a couple of dirty malformations in JSON already (trailing commas, partial parse errors) and adding a function that performs an explicit mapping for an illegal literal isn't really such a stretch. -Craig From essen@REDACTED Wed Jun 27 10:56:52 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 27 Jun 2018 10:56:52 +0200 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> <2063693.FJ7v3LWtuf@takoyaki> <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> Message-ID: <8a931674-e7be-b827-c5b5-d14679e8b691@ninenines.eu> You are conflating Javascript and JSON. JSON was originally defined as a subset of Javascript, but it has very little to do with it otherwise. There is no 'undefined' JSON type. There is only 'null'. The Javascript 'undefined' is irrelevant because it doesn't translate into JSON to begin with. That being said, both the Javascript 'undefined' and 'null' translate to the JSON 'null'. They are not treated as distinct. The JSON 'null' has roughly the same semantics as the Erlang 'undefined' so the mapping makes sense and makes using JSON libraries easier as long as the JSON library uses maps and not proplists. Cheers, On 06/27/2018 10:43 AM, Whealy, Chris wrote: > Allow me to elaborate on your point Michael (also without getting > philosophical). > > In JavaScript null?and undefined?are identifiably distinct datatypes > that serve specific purposes: > > * *null*?- This variable specifically has no value > * *undefined*?- The value of this variable is indeterminate (I.E. > identifiably different from null) > > > Although the atom null?has no special meaning in Erlang, it does when > mapped to JavaScript null; therefore to maintain accuracy, Erlang devs > who also work in JavaScript should understand and preserve this semantic > difference. ?Likewise with Erlang undefined?mapping to JavaScript undefined. > > Therefore, I submit that this semantic difference should be persevered > when mapping from Erlang to JavaScript, otherwise data loss will occur, > particularly when mapping from JavaScript back to Erlang. > > Erlang ? ?-> JavaScript > null ? ? ?-> null > undefined -> undefined > > JavaScript -> Erlang > null ? ? ? -> null > undefined ?-> undefined > > This part of the mapping table at least should be bijective. > > *Chris Whealy* > > SAP Cloud Platform | Strategy & Product Management | Team > > *SAP UK Ltd,*?Clockhouse Place, Bedfont Rd, Feltham, Middx, TW14 8HA, > England > > M +44 (0)7808 575377 > > Find out more on the Strategy & Product Management***Wiki page* > ?(SAP > internal) > Follow our latest activities in SAP CP User Community *Jam Group > * > > Please consider the impact on the environment before printing this e-mail. > > Twitter: @LogaRhythm > /"The voice of ignorance speaks loud and long,/ > /? but the words of the wise are quiet and few"/ > /? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Ancient Proverb/ > > > > > >> On 27 Jun 2018, at 08:42, Michael Nisi > > wrote: >> >> Here?s how v8::JSON, the JSON parser in Node?s JavaScript engine, does it: >> >>> JSON.stringify({}) >> '{}' >>> JSON.stringify({ name: null }) >> '{"name":null}' >>> JSON.stringify({ name: undefined }) >> '{}' >>> JSON.stringify({ name: 'Lionel' }) >> '{"name":"Lionel"}? >> >>> JSON.parse('{}').name >> undefined >>> JSON.parse('{ "name": null }').name >> null >> >> JavaScript differentiates between null and undefined, without wanting >> to get philosophical here. >> >> Michael >> >> >>> On 27. Jun 2018, at 09:21, zxq9@REDACTED wrote: >>> >>> Erlang -> JSON >>> - true ?????-> true >>> - false ????-> false >>> - undefined -> null >>> - Atom ?????-> String >>> >>> JSON -> Erlang >>> - true ?-> true >>> - false -> false >>> - null ?-> undefined >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From aseigo@REDACTED Wed Jun 27 16:19:14 2018 From: aseigo@REDACTED (Aaron Seigo) Date: Wed, 27 Jun 2018 16:19:14 +0200 Subject: [erlang-questions] term_to_binary and large data structures Message-ID: Hello! :) I have a distributed (in the Erlang sense) application which often produces moderately-sized maps (10k+ entries with lots of tuples in the mix) which in the past have given inter-node message passing serious problems: the vm would lock for a long while, use several GB of RAM, and usually eventually give up. When it didn't outright crash, it would produce message sizes too big to send between nodes, and/or the heartbeat messages between nodes would time out resulting in breakage. Running the same terms through `term_to_binary` produces similar results. The good news is that in OTP 21.0 things are quite a bit better: serialization of the maps goes a lot quicker, memory usage is now only ~500MB per encoding for terms which would quickly balloon in the multiple GB's, ... so there is progress and that is really fantastic. Is 21.0 using something other than `term_to_binary` now for inter-node messages? Is it still using the "standard" external term format? Where in the OTP source tree can one find the relevant code? Cheers! -- Aaron Seigo From zxq9@REDACTED Wed Jun 27 16:39:47 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Wed, 27 Jun 2018 23:39:47 +0900 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: References: Message-ID: <2415009.ysGbzmeS8O@takoyaki> On 2018?6?27???? 16?19?14? JST Aaron Seigo wrote: > Hello! :) > > I have a distributed (in the Erlang sense) application which often > produces moderately-sized maps (10k+ entries with lots of tuples in the > mix) which in the past have given inter-node message passing serious > problems: the vm would lock for a long while, use several GB of RAM, and > usually eventually give up. When it didn't outright crash, it would > produce message sizes too big to send between nodes, and/or the > heartbeat messages between nodes would time out resulting in breakage. > Running the same terms through `term_to_binary` produces similar > results. If you're using disterl keep in mind that inter-node heartbeats operate on the same channel as intern-node messages. If you send huge messages you are killing your heartbeats. That doesn't play out very well. The *correct* solution would be to make disterl operate over SCTP, but in reality SCTP isn't supported well enough cross-platform to make for a universal option (thanks, Microsoft). Instead, consider opening a separate TCP connection between nodes for tranfer of large data. That way heartbeats and small inter-node messages don't interfere with your huge bulk message transfers. (Yes, this is totally the kind of thing that there should be a library for, but it isn't something there is a lot of time for unpaid work on. That sucks, but there it is.) -Craig From jesper.louis.andersen@REDACTED Wed Jun 27 17:05:04 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 27 Jun 2018 17:05:04 +0200 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: References: Message-ID: The map() type now has iterators, so you can gradually iterate over the map rather than having to convert it all at once. Maybe that is what is helping you. However, I'd strongly recommend you start building up a scheme in which you chunk the large messages into smaller messages with some kind of continuation token. Large messages are bound to create trouble at some point. You can also toy with the idea of moving the code to the data rather than data to the code. On Wed, Jun 27, 2018 at 4:19 PM Aaron Seigo wrote: > Hello! :) > > I have a distributed (in the Erlang sense) application which often > produces moderately-sized maps (10k+ entries with lots of tuples in the > mix) which in the past have given inter-node message passing serious > problems: the vm would lock for a long while, use several GB of RAM, and > usually eventually give up. When it didn't outright crash, it would > produce message sizes too big to send between nodes, and/or the > heartbeat messages between nodes would time out resulting in breakage. > Running the same terms through `term_to_binary` produces similar > results. > > The good news is that in OTP 21.0 things are quite a bit better: > serialization of the maps goes a lot quicker, memory usage is now only > ~500MB per encoding for terms which would quickly balloon in the > multiple GB's, ... so there is progress and that is really fantastic. > > Is 21.0 using something other than `term_to_binary` now for inter-node > messages? > > Is it still using the "standard" external term format? > > Where in the OTP source tree can one find the relevant code? > > Cheers! > > -- > Aaron Seigo > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aseigo@REDACTED Wed Jun 27 23:24:35 2018 From: aseigo@REDACTED (Aaron Seigo) Date: Wed, 27 Jun 2018 23:24:35 +0200 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: References: Message-ID: On 2018-06-27 17:05, Jesper Louis Andersen wrote: > The map() type now has iterators, so you can gradually iterate over the > map rather > than having to convert it all at once. Maybe that is what is helping > you. That could well be it. > However, I'd strongly recommend you start building up a scheme in which > you chunk the > large messages into smaller messages with some kind of continuation > token. We already do. This does not, however, resolve the real issue which is bandwidth usage. Chunking the messages just makes smaller bits of bloat. The total bloat is exactly the same, however, and easily inundates Gbit and even 10Gbit networking. Except now we have the _added overhead_ of more messages. It's merely a way to shuffle forward, not a path to anything scalable. > Large messages are bound to create trouble at some point. Yes, if unbounded, I would agree. However, that is not our case. We have maps with 10k keys that strain this system and easily saturate our network. This is not "big" by any modern definition. As a demonstration of this to ourselves, I wrote an Elixir library that serializes terms to a more space efficient format. Where `term_to_binary` creates 500MB monsters, this library conveniently creates a 1.5MB binary out of the exact same data. In fact, for nearly any term you throw at it, this pretty simple algorithm produces smaller serialized data. You can see the format employed here: https://github.com/aseigo/packer/blob/develop/FORMAT.md Given that it routinely produces results anywhere from 33% to 99% (!!) smaller just shows how problematic the current external term format is. Unfortunately, this is "only" an Elixir implementation and so is not very fast at this point. The point of the exercise was to see what a reasonable term serializer could produce, specifically to see if there was any improvement to be had. Apparently there is quite a bit. The external term format is widely used, so improvements in it could be far reaching. How difficult would it be to change the external term format, based on e.g. the versioning in the distribution header? Would it be possible to make term serialization pluggable as more and more of the the rest of the distribution framework in the BEAM has become in v 21? > You can also toy with the > idea of moving the code to the data rather than data to the code. Our goal is to distribute computation, so that would be counter-productive. -- Aaron From aseigo@REDACTED Wed Jun 27 23:33:45 2018 From: aseigo@REDACTED (Aaron Seigo) Date: Wed, 27 Jun 2018 23:33:45 +0200 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: <2415009.ysGbzmeS8O@takoyaki> References: <2415009.ysGbzmeS8O@takoyaki> Message-ID: On 2018-06-27 16:39, zxq9@REDACTED wrote: > On 2018?6?27???? 16?19?14? JST Aaron Seigo wrote: > If you send huge messages you are killing your heartbeats. That doesn't > play out very well. Yes, we've noticed ;) > Instead, consider opening a separate TCP connection between nodes for > tranfer of large data. That way heartbeats and small inter-node > messages > don't interfere with your huge bulk message transfers. That would be fine, and quite doable ... but it means everyone having similar issues needs to do the same. Why are we using Erlang, again? Yes, for ease of use of things like concurrency and distribution. When those things break, the reason to use Erlang evaporates. It would be better to address the issues (even if it takes a while and some effort to do so!) then to work around it. That said .. the recent work on top of 21 to allow out-of-order sending of messages, including during chunked message transmission, to avoid head-of-line blocking is really promising and should, once finalized and upstreamed, resolve this particular issue. > (Yes, this is totally the kind of thing that there should be a library > for, but it isn't something there is a lot of time for unpaid work on. > That sucks, but there it is.) IMHO it should be part of the distribution system itself, one way or the other, as using a library to work *around* distribution (and which itself would need to follow distribution to be transparent) is a little daft. There are enough of us working on paid time these days to make it happen, as long as we know what needs doing and what is acceptable. :) -- Aaron From chandrashekhar.mullaparthi@REDACTED Wed Jun 27 23:34:36 2018 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 27 Jun 2018 22:34:36 +0100 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: <2415009.ysGbzmeS8O@takoyaki> References: <2415009.ysGbzmeS8O@takoyaki> Message-ID: On 27 June 2018 at 15:39, wrote: > On 2018?6?27???? 16?19?14? JST Aaron Seigo wrote: > > Hello! :) > > > > I have a distributed (in the Erlang sense) application which often > > produces moderately-sized maps (10k+ entries with lots of tuples in the > > mix) which in the past have given inter-node message passing serious > > problems: the vm would lock for a long while, use several GB of RAM, and > > usually eventually give up. When it didn't outright crash, it would > > produce message sizes too big to send between nodes, and/or the > > heartbeat messages between nodes would time out resulting in breakage. > > Running the same terms through `term_to_binary` produces similar > > results. > > If you're using disterl keep in mind that inter-node heartbeats operate > on the same channel as intern-node messages. > > If you send huge messages you are killing your heartbeats. That doesn't > play out very well. > > The *correct* solution would be to make disterl operate over SCTP, but in > reality SCTP isn't supported well enough cross-platform to make for a > universal option (thanks, Microsoft). > > Instead, consider opening a separate TCP connection between nodes for > tranfer of large data. That way heartbeats and small inter-node messages > don't interfere with your huge bulk message transfers. > > (Yes, this is totally the kind of thing that there should be a library > for, but it isn't something there is a lot of time for unpaid work on. > That sucks, but there it is.) > > There are a few libraries which offer this functionality. Here are two I know of. https://github.com/priestjim/gen_rpc https://github.com/Bluehouse-Technology/erpc Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Wed Jun 27 23:57:13 2018 From: t@REDACTED (Tristan Sloughter) Date: Wed, 27 Jun 2018 15:57:13 -0600 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: References: <2415009.ysGbzmeS8O@takoyaki> Message-ID: <1530136633.735265.1422741168.0D66C9BB@webmail.messagingengine.com> > That said .. the recent work on top of 21 to allow out-of-order sending > of messages, including during chunked message transmission, to avoid > head-of-line blocking is really promising and should, once finalized and > upstreamed, resolve this particular issue. Yes, this is going to be great. And with the changes internally for 21 I was able to get my SCTP experiment working, https://github.com/tsloughter/sctp_dist From zxq9@REDACTED Wed Jun 27 23:58:31 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 28 Jun 2018 06:58:31 +0900 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: <1530136633.735265.1422741168.0D66C9BB@webmail.messagingengine.com> References: <1530136633.735265.1422741168.0D66C9BB@webmail.messagingengine.com> Message-ID: <18436968.ZNrRiElmWW@takoyaki> On 2018?6?27???? 15?57?13? JST Tristan Sloughter wrote: > > That said .. the recent work on top of 21 to allow out-of-order sending > > of messages, including during chunked message transmission, to avoid > > head-of-line blocking is really promising and should, once finalized and > > upstreamed, resolve this particular issue. > > Yes, this is going to be great. And with the changes internally for 21 I > was able to get my SCTP experiment working, > https://github.com/tsloughter/sctp_dist It's like... the future! -Craig From mononcqc@REDACTED Thu Jun 28 01:14:10 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 27 Jun 2018 19:14:10 -0400 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: References: Message-ID: <20180627231409.GB95749@ferdmbp.local> On 06/27, Aaron Seigo wrote: >We have maps with 10k keys that strain this system and easily saturate >our network. This is not "big" by any modern definition. As a >demonstration of this to ourselves, I wrote an Elixir library that >serializes terms to a more space efficient format. Where >`term_to_binary` creates 500MB monsters, this library conveniently >creates a 1.5MB binary out of the exact same data. > Have you tried comparing when `term_to_binary(Term, [{compressed, 9}])'? If you can pack 500MB of data down to 1.5 MB, chances are that compression could do some good things on your end. Regards, Fred. From zxq9@REDACTED Thu Jun 28 02:32:03 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 28 Jun 2018 09:32:03 +0900 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: <20180627231409.GB95749@ferdmbp.local> References: <20180627231409.GB95749@ferdmbp.local> Message-ID: <1801757.4jeNc9cPXT@takoyaki> On 2018?6?27???? 19?14?10? JST Fred Hebert wrote: > On 06/27, Aaron Seigo wrote: > >We have maps with 10k keys that strain this system and easily saturate > >our network. This is not "big" by any modern definition. As a > >demonstration of this to ourselves, I wrote an Elixir library that > >serializes terms to a more space efficient format. Where > >`term_to_binary` creates 500MB monsters, this library conveniently > >creates a 1.5MB binary out of the exact same data. > > > > Have you tried comparing when `term_to_binary(Term, [{compressed, 9}])'? > If you can pack 500MB of data down to 1.5 MB, chances are that > compression could do some good things on your end. With ratios like that it is also highly likely that the bulk of the data is not actually information... -Craig From mjtruog@REDACTED Thu Jun 28 02:45:51 2018 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 27 Jun 2018 17:45:51 -0700 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: References: Message-ID: <8b22a68f-ab45-c805-ddc8-a21abb050ae0@gmail.com> On 06/27/2018 07:19 AM, Aaron Seigo wrote: > I have a distributed (in the Erlang sense) application which often produces moderately-sized maps (10k+ entries with lots of tuples in the mix) which in the past have given inter-node message passing serious problems: the vm would lock for a long while, use several GB of RAM, and usually eventually give up. When it didn't outright crash, it would produce message sizes too big to send between nodes, and/or the heartbeat messages between nodes would time out resulting in breakage. Running the same terms through `term_to_binary` produces similar results. > > The good news is that in OTP 21.0 things are quite a bit better: serialization of the maps goes a lot quicker, memory usage is now only ~500MB per encoding for terms which would quickly balloon in the multiple GB's, ... so there is progress and that is really fantastic. > Part of what you may be seeing is the amount of memory allocated for the receiver of a distributed Erlang message that contains a large Erlang map because of the need to over-estimate the total size of the map at https://github.com/erlang/otp/blob/f3790140d0e73f257c78d67de894b606ef53a8e5/erts/emulator/beam/erl_map.h#L196-L197 (not sure if other related logic changed recently, others may want to comment on that, if that is the case). Your messages sound big enough that you may want to consider switching to a less-dynamic binary format, if that is possible with your usage of the data, to minimize the potential memory consumption.? Get/Put operations on a binary are very slow (e.g., at https://github.com/okeuday/blookup) though it may help you deal with the memory usage in a simpler way. Best Regards, Michael From vans_163@REDACTED Thu Jun 28 04:25:21 2018 From: vans_163@REDACTED (Vans S) Date: Thu, 28 Jun 2018 02:25:21 +0000 (UTC) Subject: [erlang-questions] Is there a way to disable EPMD in Erlang releases? References: <2043113197.3712104.1530152721219.ref@mail.yahoo.com> Message-ID: <2043113197.3712104.1530152721219@mail.yahoo.com> Hello, ? ? I am currently compiling an Erlang release of a PoC app that is ending up being used on Win Mac and Linux.? Is there any way to tell the release to not include and/or not start EPMD? The reason is the app does not use distributed erlang at all, and on windows it results in extra firewall exceptions being triggered. On linux it results in systemd thinking the process tree is alive if EPMD was started by the release then the release dies/was stopped. -------------- next part -------------- An HTML attachment was scrubbed... URL: From be.dmitry@REDACTED Thu Jun 28 06:49:49 2018 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Thu, 28 Jun 2018 14:49:49 +1000 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <8a931674-e7be-b827-c5b5-d14679e8b691@ninenines.eu> References: <1567611.nGgtL7t0W2@takoyaki> <2063693.FJ7v3LWtuf@takoyaki> <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> <8a931674-e7be-b827-c5b5-d14679e8b691@ninenines.eu> Message-ID: I could never understand where this conversion JSON null <-> Erlang undefined was coming from. Of course we generally don't use null atom in the code, but I always considered "undefined" value in Erlang meaning "absent" value. For example when a record is created and some field is not populated explicitly, it is set undefined. This logic is very similar to Javascript objects with missing fields. Consider Javascript parsing a JSON object and fetching fields from it. If a key was not present in the JSON, undefined would be returned, which is different from null. That example always made me think, that undefined values on Erlang side should be seen on Javascript side as undefined values too. And to make sure this works that way, the attributes holding atom undefined had to be not present in the resulting JSON. There is another example from Javascript world: > JSON.stringify({a: 1, b: undefined, c: null}) < "{"a":1,"c":null}" So I'd say that to keep semantics synchronised, the conversions should be JSON null <-> Erlang null, and Erlang undefined values in objects must be simply removed from the resulting JSON object. Anyway, what I'd actually greet as a serious improvement in handling of JSON in our small Erlang world would be an introduction of "SAX JSON" parser (similar to what jsx does) as part of the standard Erlang distribution and implemented in C. That way we would have parsing separated from construction of the representation, and anyone would be able to use whatever representation they like. Kind regards, Dmitry Belyaev On Wed, Jun 27, 2018 at 6:56 PM, Lo?c Hoguin wrote: > You are conflating Javascript and JSON. JSON was originally defined as a > subset of Javascript, but it has very little to do with it otherwise. > > There is no 'undefined' JSON type. There is only 'null'. The Javascript > 'undefined' is irrelevant because it doesn't translate into JSON to begin > with. > > That being said, both the Javascript 'undefined' and 'null' translate to > the JSON 'null'. They are not treated as distinct. The JSON 'null' has > roughly the same semantics as the Erlang 'undefined' so the mapping makes > sense and makes using JSON libraries easier as long as the JSON library > uses maps and not proplists. > > Cheers, > > On 06/27/2018 10:43 AM, Whealy, Chris wrote: > >> Allow me to elaborate on your point Michael (also without getting >> philosophical). >> >> In JavaScript null and undefined are identifiably distinct datatypes that >> serve specific purposes: >> >> * *null* - This variable specifically has no value >> * *undefined* - The value of this variable is indeterminate (I.E. >> identifiably different from null) >> >> >> Although the atom null has no special meaning in Erlang, it does when >> mapped to JavaScript null; therefore to maintain accuracy, Erlang devs who >> also work in JavaScript should understand and preserve this semantic >> difference. Likewise with Erlang undefined mapping to JavaScript undefined. >> >> Therefore, I submit that this semantic difference should be persevered >> when mapping from Erlang to JavaScript, otherwise data loss will occur, >> particularly when mapping from JavaScript back to Erlang. >> >> Erlang -> JavaScript >> null -> null >> undefined -> undefined >> >> JavaScript -> Erlang >> null -> null >> undefined -> undefined >> >> This part of the mapping table at least should be bijective. >> >> *Chris Whealy* >> >> SAP Cloud Platform | Strategy & Product Management | Team >> >> *SAP UK Ltd,* Clockhouse Place, Bedfont Rd, Feltham, Middx, TW14 8HA, >> England >> >> M +44 (0)7808 575377 >> >> Find out more on the Strategy & Product Management***Wiki page* < >> https://wiki.wdf.sap.corp/wiki/pages/viewpage.action?pageId=1865737441> (SAP >> internal) >> Follow our latest activities in SAP CP User Community *Jam Group < >> https://jam4.sapjam.com/groups/about_page/eopqUq5S182gY7JFrbdwis>* >> >> Please consider the impact on the environment before printing this e-mail. >> >> Twitter: @LogaRhythm >> /"The voice of ignorance speaks loud and long,/ >> / but the words of the wise are quiet and few"/ >> / Ancient Proverb/ >> >> >> >> >> >> On 27 Jun 2018, at 08:42, Michael Nisi >> michael.nisi@REDACTED>> wrote: >>> >>> Here?s how v8::JSON, the JSON parser in Node?s JavaScript engine, does >>> it: >>> >>> JSON.stringify({}) >>>> >>> '{}' >>> >>>> JSON.stringify({ name: null }) >>>> >>> '{"name":null}' >>> >>>> JSON.stringify({ name: undefined }) >>>> >>> '{}' >>> >>>> JSON.stringify({ name: 'Lionel' }) >>>> >>> '{"name":"Lionel"}? >>> >>> JSON.parse('{}').name >>>> >>> undefined >>> >>>> JSON.parse('{ "name": null }').name >>>> >>> null >>> >>> JavaScript differentiates between null and undefined, without wanting to >>> get philosophical here. >>> >>> Michael >>> >>> >>> On 27. Jun 2018, at 09:21, zxq9@REDACTED wrote: >>>> >>>> Erlang -> JSON >>>> - true -> true >>>> - false -> false >>>> - undefined -> null >>>> - Atom -> String >>>> >>>> JSON -> Erlang >>>> - true -> true >>>> - false -> false >>>> - null -> undefined >>>> >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > https://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aseigo@REDACTED Thu Jun 28 07:16:48 2018 From: aseigo@REDACTED (Aaron Seigo) Date: Thu, 28 Jun 2018 07:16:48 +0200 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: <20180627231409.GB95749@ferdmbp.local> References: <20180627231409.GB95749@ferdmbp.local> Message-ID: On 2018-06-28 01:14, Fred Hebert wrote: > On 06/27, Aaron Seigo wrote: >> We have maps with 10k keys that strain this system and easily saturate >> our network. This is not "big" by any modern definition. As a >> demonstration of this to ourselves, I wrote an Elixir library that >> serializes terms to a more space efficient format. Where >> `term_to_binary` creates 500MB monsters, this library conveniently >> creates a 1.5MB binary out of the exact same data. >> > > Have you tried comparing when `term_to_binary(Term, [{compressed, > 9}])'? If you can pack 500MB of data down to 1.5 MB, chances are that > compression could do some good things on your end. Yes, and it certainly helps but it is still larger than one would hope for (and larger than what that POC produces), but most importantly this only is meaningful when we control the call to `term_to_binary`. When it is hidden behind code in OTP or a library, or an equivalent function is generating an external term format binary, we don't get to use this trick. Which also brings us to the fact that the compression being used is still zlib, while there are much better options out there. That POC implementation uses zstd which is both faster and produces smaller binaries than zlib. -- Aaron From aseigo@REDACTED Thu Jun 28 07:25:12 2018 From: aseigo@REDACTED (Aaron Seigo) Date: Thu, 28 Jun 2018 07:25:12 +0200 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: <8b22a68f-ab45-c805-ddc8-a21abb050ae0@gmail.com> References: <8b22a68f-ab45-c805-ddc8-a21abb050ae0@gmail.com> Message-ID: On 2018-06-28 02:45, Michael Truog wrote: > On 06/27/2018 07:19 AM, Aaron Seigo wrote: >> I have a distributed (in the Erlang sense) application which often >> produces moderately-sized maps (10k+ entries with lots of tuples in >> the mix) which in the past have given inter-node message passing >> serious problems: the vm would lock for a long while, use several GB >> of RAM, and usually eventually give up. When it didn't outright crash, >> it would produce message sizes too big to send between nodes, and/or >> the heartbeat messages between nodes would time out resulting in >> breakage. Running the same terms through `term_to_binary` produces >> similar results. >> >> The good news is that in OTP 21.0 things are quite a bit better: >> serialization of the maps goes a lot quicker, memory usage is now only >> ~500MB per encoding for terms which would quickly balloon in the >> multiple GB's, ... so there is progress and that is really fantastic. >> > > Part of what you may be seeing is the amount of memory allocated for > the receiver of a distributed Erlang message that contains a large > Erlang map because of the need to over-estimate the total size of the > map at Those numbers were all from the sending side; the maps don't seem to be an issue on deserialization. > messages sound big enough that you may want to consider switching > to a less-dynamic binary format, Everything is possible ;) but not everything is palatable. The maps are generated in part by NIFs so stepping outside the standard set of data structures becomes more difficult, and for our use cases maps are the "right" data structure not just to represent the data but more importantly to work with it. I'm not a fan of working around a problem when the cause and location of it is easily noted. It would be much nicer to improve the serialization of data for messages, not only for our needs, but since it would positively impact every user of the BEAM for distribution. Thanks for the pointer to blookup, though; neat approach. We don't really have the issue of usage between processes, though, as much as we do between nodes. So reference counting can't really help us :) -- Aaron From zxq9@REDACTED Thu Jun 28 07:27:02 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 28 Jun 2018 14:27:02 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> <8a931674-e7be-b827-c5b5-d14679e8b691@ninenines.eu> Message-ID: <6742394.3zBt6amhjZ@takoyaki> On 2018?6?28???? 14?49?49? JST Dmitry Belyaev wrote: > Anyway, what I'd actually greet as a serious improvement in handling of > JSON in our small Erlang world would be an introduction of "SAX JSON" > parser (similar to what jsx does) as part of the standard Erlang > distribution and implemented in C. That way we would have parsing separated > from construction of the representation, and anyone would be able to use > whatever representation they like. We've discussed that more than a few times. The problem is that there is no "right" mapping because JSON, lacking almost all useful types, is extremely lossy. JSX handles objects as proplists in encoding and decoding, but can decode them as maps if you tell it to. ZJ handles maps objects to Erlang maps directly, and tuples to lists, meaning you a proplist encoded to JSON becomes a list of ordered pairs. Which one is right? They both are, technically speaking, but one or the other is going to suit a given project better than the other -- and as a library author there is no way to know which way will suit project X better ahead of time. zj:encode/1 and zj:decode/1 are list-based -- which leaves some ambiguity but works better in *most* cases. zj:binary_encode/1 and zj:binary_decode/1 is strict about strings being binaries and lists being lists, closer to the way JSX works. Which one is right? That's not possible to know. Fortunately we live in a world where using JSX or ZJ or Jiffy (if your platform can build C, anyway) or writing your own isn't hard, so we get more options this way. I think that's a LOT better situation than burdening the core distribution team with trying to pick the "right" way, that falling through and instead having to maintain a zillion optional arguments for encode and decode, maintaining a portable C codebase, and dealing with all the complaints that will inevitably result from there ever having been a default chosen in advance (whatever defaults are innocently chosen for encode/1 and decode/1 will surely be wrong in the view of over half the users out there). I really like that Erlang is a "batteries included" language, but thre are clear limits. Consider how many projects can't use httpc, the built-in web parsers, or similar tools. How much other work could be done were the maintenance burden of those bits removed? Not to mention backward compatibile maintenance being a tarbaby all its own... -Craig From aseigo@REDACTED Thu Jun 28 07:36:36 2018 From: aseigo@REDACTED (Aaron Seigo) Date: Thu, 28 Jun 2018 07:36:36 +0200 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: References: Message-ID: <131a8c8d06640db106b71fd1ef246fe7@mykolab.com> On 2018-06-28 07:03, Dmitry Belyaev wrote: > So it's not so bad as it's stated in the file, only 33 time worse than > the advertised > format. My bad; this came from a run of performance tests where the size of them map is increased incrementally. It is missing a zero in one of the lines; will fix. > However after applying [compressed: 9] suggested by Fred Hebert, I see: > > iex(6)> :erlang.term_to_binary(tuple_map, compressed: 9) |> byte_size() > |> (fn x -> x / 1024 / 1024 end).() > 0.38570117950439453 There are three problems with this: a) that data does get decompressed at some point. It doesn't magically go away. It does, however, help with network usage. b) we trade time (to compress) for space (fewer byte to transmit), when that space isn't needed in the first place. The time required to compress, esp relative to the serialization time, is decidedly non-trivial. It takes several times as long to compress with zlib than it does to serialize. This could be improved by moving a more modern compression algorithm, though the cost is always non-zero of course. In our tests, it definitely paid to compress the actual data in the map, but there was very little need to compress the structural metadata when encoded efficiently. c) we don't always control the call to term_to_binary, or the equivalent eternal term generators, and so don't have access to compression e.g. on distribution messages I suppose we could propose using compression on (larger) distribution messages, which would help with the network saturation, and would be a better stop-gap than nothing, but it still leaves us with (a) and (b) above (and , and (c) everywhere else. -- Aaron From essen@REDACTED Thu Jun 28 09:31:33 2018 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 28 Jun 2018 09:31:33 +0200 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> <2063693.FJ7v3LWtuf@takoyaki> <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> <8a931674-e7be-b827-c5b5-d14679e8b691@ninenines.eu> Message-ID: Again I'm questioning why you insist on comparing it to Javascript. JSON is used with pretty much all languages and there is no 1:1 language-JSON mapping that I know of, including Javascript. JSON is a language-agnostic serialization format. JSON does not have the Javascript undefined. The only way to get the Javascript undefined out of JSON is by not defining a JSON object's field at all. In Erlang we do this by not defining the map field at all. JSON has null, it corresponds to the Javascript null and is pretty much what we use undefined for in Erlang. In maps it represents a field that has a key defined but no value. This library uses maps for objects. Not records, not proplists, so there's no pernicious issue with implicit undefined (records' default undefined or proplists:get_value's default undefined). All undefined values are set explicitly by the user to mean the same as null. I do not see an issue with that. Cheers, On 06/28/2018 06:49 AM, Dmitry Belyaev wrote: > I could never understand where this conversion JSON null <-> Erlang > undefined was coming from. > > Of course we generally don't use null atom in the code, but I always > considered "undefined" value in Erlang meaning "absent" value. For > example when a record is created and some field is not populated > explicitly, it is set undefined. > This logic is very similar to Javascript objects with missing fields. > Consider Javascript parsing a JSON object and fetching fields from it. > If a key was not present in the JSON, undefined would be returned, which > is different from null. > > That example always made me think, that undefined values on Erlang side > should be seen on Javascript side as undefined values too. And to make > sure this works that way, the attributes holding atom undefined had to > be not present in the resulting JSON. > > There is another example from Javascript world: > > JSON.stringify({a: 1, b: undefined, c: null}) > < "{"a":1,"c":null}" > > So I'd say that to keep semantics synchronised, the conversions should > be JSON null <-> Erlang null, and Erlang undefined values in objects > must be simply removed from the resulting JSON object. > > Anyway, what I'd actually greet as a serious improvement in handling of > JSON in our small Erlang world would be an introduction of "SAX JSON" > parser (similar to what jsx does) as part of the standard Erlang > distribution and implemented in C. That way we would have parsing > separated from construction of the representation, and anyone would be > able to use whatever representation they like. > > Kind regards, > Dmitry Belyaev > > On Wed, Jun 27, 2018 at 6:56 PM, Lo?c Hoguin > wrote: > > You are conflating Javascript and JSON. JSON was originally defined > as a subset of Javascript, but it has very little to do with it > otherwise. > > There is no 'undefined' JSON type. There is only 'null'. The > Javascript 'undefined' is irrelevant because it doesn't translate > into JSON to begin with. > > That being said, both the Javascript 'undefined' and 'null' > translate to the JSON 'null'. They are not treated as distinct. The > JSON 'null' has roughly the same semantics as the Erlang 'undefined' > so the mapping makes sense and makes using JSON libraries easier as > long as the JSON library uses maps and not proplists. > > Cheers, > > On 06/27/2018 10:43 AM, Whealy, Chris wrote: > > Allow me to elaborate on your point Michael (also without > getting philosophical). > > In JavaScript null?and undefined?are identifiably distinct > datatypes that serve specific purposes: > > ? * *null*?- This variable specifically has no value > ? * *undefined*?- The value of this variable is indeterminate (I.E. > ? ? identifiably different from null) > > > Although the atom null?has no special meaning in Erlang, it does > when mapped to JavaScript null; therefore to maintain accuracy, > Erlang devs who also work in JavaScript should understand and > preserve this semantic difference.? Likewise with Erlang > undefined?mapping to JavaScript undefined. > > Therefore, I submit that this semantic difference should be > persevered when mapping from Erlang to JavaScript, otherwise > data loss will occur, particularly when mapping from JavaScript > back to Erlang. > > Erlang ? ?-> JavaScript > null ? ? ?-> null > undefined -> undefined > > JavaScript -> Erlang > null ? ? ? -> null > undefined ?-> undefined > > This part of the mapping table at least should be bijective. > > *Chris Whealy* > > SAP Cloud Platform | Strategy & Product Management | Team > > *SAP UK Ltd,*?Clockhouse Place, Bedfont Rd, Feltham, Middx, TW14 > 8HA, England > > M +44 (0)7808 575377 > > Find out more on the Strategy & Product Management***Wiki page* > >?(SAP > internal) > Follow our latest activities in SAP CP User Community *Jam Group > >* > > Please consider the impact on the environment before printing > this e-mail. > > Twitter: @LogaRhythm > /"The voice of ignorance speaks loud and long,/ > /? but the words of the wise are quiet and few"/ > /? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Ancient Proverb/ > > > > > > On 27 Jun 2018, at 08:42, Michael Nisi > > >> wrote: > > Here?s how v8::JSON, the JSON parser in Node?s JavaScript > engine, does it: > > JSON.stringify({}) > > '{}' > > JSON.stringify({ name: null }) > > '{"name":null}' > > JSON.stringify({ name: undefined }) > > '{}' > > JSON.stringify({ name: 'Lionel' }) > > '{"name":"Lionel"}? > > JSON.parse('{}').name > > undefined > > JSON.parse('{ "name": null }').name > > null > > JavaScript differentiates between null and undefined, > without wanting to get philosophical here. > > Michael > > > On 27. Jun 2018, at 09:21, zxq9@REDACTED > > wrote: > > Erlang -> JSON > - true ?????-> true > - false ????-> false > - undefined -> null > - Atom ?????-> String > > JSON -> Erlang > - true ?-> true > - false -> false > - null ?-> undefined > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > https://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From oltarasenko@REDACTED Thu Jun 28 09:46:10 2018 From: oltarasenko@REDACTED (Oleg Tarasenko) Date: Thu, 28 Jun 2018 09:46:10 +0200 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> <2063693.FJ7v3LWtuf@takoyaki> <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> <8a931674-e7be-b827-c5b5-d14679e8b691@ninenines.eu> Message-ID: Hi Craig, The lib looks interesting, I would like to try it on my projects. Few related questions: 1) Do you plan to add tests later on? 2) What do you think about publishing it on hex.pm, so it's easier to control releases (for those who are using it), maybe rebar3/erlang.mk can be quite helpful here? 3) Finally, the performance, did you have a chance to make some comparisons? Best regards, Oleg On Thu, Jun 28, 2018 at 9:31 AM Lo?c Hoguin wrote: > Again I'm questioning why you insist on comparing it to Javascript. JSON > is used with pretty much all languages and there is no 1:1 language-JSON > mapping that I know of, including Javascript. JSON is a > language-agnostic serialization format. > > JSON does not have the Javascript undefined. The only way to get the > Javascript undefined out of JSON is by not defining a JSON object's > field at all. In Erlang we do this by not defining the map field at all. > > JSON has null, it corresponds to the Javascript null and is pretty much > what we use undefined for in Erlang. In maps it represents a field that > has a key defined but no value. > > This library uses maps for objects. Not records, not proplists, so > there's no pernicious issue with implicit undefined (records' default > undefined or proplists:get_value's default undefined). All undefined > values are set explicitly by the user to mean the same as null. I do not > see an issue with that. > > Cheers, > > On 06/28/2018 06:49 AM, Dmitry Belyaev wrote: > > I could never understand where this conversion JSON null <-> Erlang > > undefined was coming from. > > > > Of course we generally don't use null atom in the code, but I always > > considered "undefined" value in Erlang meaning "absent" value. For > > example when a record is created and some field is not populated > > explicitly, it is set undefined. > > This logic is very similar to Javascript objects with missing fields. > > Consider Javascript parsing a JSON object and fetching fields from it. > > If a key was not present in the JSON, undefined would be returned, which > > is different from null. > > > > That example always made me think, that undefined values on Erlang side > > should be seen on Javascript side as undefined values too. And to make > > sure this works that way, the attributes holding atom undefined had to > > be not present in the resulting JSON. > > > > There is another example from Javascript world: > > > JSON.stringify({a: 1, b: undefined, c: null}) > > < "{"a":1,"c":null}" > > > > So I'd say that to keep semantics synchronised, the conversions should > > be JSON null <-> Erlang null, and Erlang undefined values in objects > > must be simply removed from the resulting JSON object. > > > > Anyway, what I'd actually greet as a serious improvement in handling of > > JSON in our small Erlang world would be an introduction of "SAX JSON" > > parser (similar to what jsx does) as part of the standard Erlang > > distribution and implemented in C. That way we would have parsing > > separated from construction of the representation, and anyone would be > > able to use whatever representation they like. > > > > Kind regards, > > Dmitry Belyaev > > > > On Wed, Jun 27, 2018 at 6:56 PM, Lo?c Hoguin > > wrote: > > > > You are conflating Javascript and JSON. JSON was originally defined > > as a subset of Javascript, but it has very little to do with it > > otherwise. > > > > There is no 'undefined' JSON type. There is only 'null'. The > > Javascript 'undefined' is irrelevant because it doesn't translate > > into JSON to begin with. > > > > That being said, both the Javascript 'undefined' and 'null' > > translate to the JSON 'null'. They are not treated as distinct. The > > JSON 'null' has roughly the same semantics as the Erlang 'undefined' > > so the mapping makes sense and makes using JSON libraries easier as > > long as the JSON library uses maps and not proplists. > > > > Cheers, > > > > On 06/27/2018 10:43 AM, Whealy, Chris wrote: > > > > Allow me to elaborate on your point Michael (also without > > getting philosophical). > > > > In JavaScript null and undefined are identifiably distinct > > datatypes that serve specific purposes: > > > > * *null* - This variable specifically has no value > > * *undefined* - The value of this variable is indeterminate > (I.E. > > identifiably different from null) > > > > > > Although the atom null has no special meaning in Erlang, it does > > when mapped to JavaScript null; therefore to maintain accuracy, > > Erlang devs who also work in JavaScript should understand and > > preserve this semantic difference. Likewise with Erlang > > undefined mapping to JavaScript undefined. > > > > Therefore, I submit that this semantic difference should be > > persevered when mapping from Erlang to JavaScript, otherwise > > data loss will occur, particularly when mapping from JavaScript > > back to Erlang. > > > > Erlang -> JavaScript > > null -> null > > undefined -> undefined > > > > JavaScript -> Erlang > > null -> null > > undefined -> undefined > > > > This part of the mapping table at least should be bijective. > > > > *Chris Whealy* > > > > SAP Cloud Platform | Strategy & Product Management | Team > > > > *SAP UK Ltd,* Clockhouse Place, Bedfont Rd, Feltham, Middx, TW14 > > 8HA, England > > > > M +44 (0)7808 575377 > > > > Find out more on the Strategy & Product Management***Wiki page* > > < > https://wiki.wdf.sap.corp/wiki/pages/viewpage.action?pageId=1865737441 > > < > https://wiki.wdf.sap.corp/wiki/pages/viewpage.action?pageId=1865737441 > >> (SAP > > internal) > > Follow our latest activities in SAP CP User Community *Jam Group > > < > https://jam4.sapjam.com/groups/about_page/eopqUq5S182gY7JFrbdwis < > https://jam4.sapjam.com/groups/about_page/eopqUq5S182gY7JFrbdwis>>* > > > > Please consider the impact on the environment before printing > > this e-mail. > > > > Twitter: @LogaRhythm > > /"The voice of ignorance speaks loud and long,/ > > / but the words of the wise are quiet and few"/ > > / Ancient > Proverb/ > > > > > > > > > > > > On 27 Jun 2018, at 08:42, Michael Nisi > > > > > >> wrote: > > > > Here?s how v8::JSON, the JSON parser in Node?s JavaScript > > engine, does it: > > > > JSON.stringify({}) > > > > '{}' > > > > JSON.stringify({ name: null }) > > > > '{"name":null}' > > > > JSON.stringify({ name: undefined }) > > > > '{}' > > > > JSON.stringify({ name: 'Lionel' }) > > > > '{"name":"Lionel"}? > > > > JSON.parse('{}').name > > > > undefined > > > > JSON.parse('{ "name": null }').name > > > > null > > > > JavaScript differentiates between null and undefined, > > without wanting to get philosophical here. > > > > Michael > > > > > > On 27. Jun 2018, at 09:21, zxq9@REDACTED > > > > wrote: > > > > Erlang -> JSON > > - true -> true > > - false -> false > > - undefined -> null > > - Atom -> String > > > > JSON -> Erlang > > - true -> true > > - false -> false > > - null -> undefined > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > -- > > Lo?c Hoguin > > https://ninenines.eu > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Jun 28 10:16:47 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 28 Jun 2018 17:16:47 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> Message-ID: <382545037.n1KB41SD1S@takoyaki> On 2018?6?28???? 9?46?10? JST you wrote: > Hi Craig, > > The lib looks interesting, I would like to try it on my projects. > > Few related questions: > 1) Do you plan to add tests later on? > 2) What do you think about publishing it on hex.pm, so it's easier > to control releases (for those who are using it), maybe rebar3/erlang.mk > can be quite helpful here? > 3) Finally, the performance, did you have a chance to make some comparisons? Hi, Oleg! 1- I fuzzed it by running real-world JSON through it, but am not convinced of the utility of hand-written tests considering how narrow the actual spec is compared to the variety of real-world illegal JSON out there. I think the best way to *really* approach testing would be to write a script that would evaluate its performance against either a bulk dataset stored for that purpose or (much better) use PropEr. I'm open to that, but don't have the time right now. It works for my cases exactly as I intended, but I'm certainly open to contributions. I fully realize my position is heretical. 2- If someone wants to push it to hex.pm that would be swell! I don't use hex.pm in any projects at the moment, so it is a very low priority. Again, more of a time issue for me than anything. My "extra" time budget wound up getting spent on making sure the docs came out properly. Same as #1, I'm open to that and would love to see it packaged for others to use in a more simple way -- but it isn't a personal priority for me at the moment. It will, however, be part of Zomp's default FOSS repository once I get around to releasing that (so will JSX). 3- The only function I benchmarked formally was decode/1. It performed about 5% (+/- 10) faster than JSX decode/1 -- so for all practical purposes is the same. The *size* of the output of decode/1 is larger than JSX when dealing with strings because decode/1 returns strings-as-strings, not binaries. This is better for some kinds of manipulation, but isn't very compact. binary_decode/1 almost certainly performs a little slower than decode/1 because I didn't implement that in a particularly efficient way. I can improve it to a comparable speed if that becomes necessary, though. Anyway, this has not been benchmarked formally, just poked at with timer:tc/1 a few times (but that isn't a particularly reliable way to benchmark things). I have not benchmarked encode/1 or binary_encode/1 at all, but would be very surprised if they are meaningfully slower than JSX as the approach taken is very similar -- the mapping between JSON <=> Erlang types being the important difference. Nice to hear from you! -Craig From michal@REDACTED Thu Jun 28 10:44:13 2018 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Thu, 28 Jun 2018 10:44:13 +0200 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <382545037.n1KB41SD1S@takoyaki> References: <1567611.nGgtL7t0W2@takoyaki> <382545037.n1KB41SD1S@takoyaki> Message-ID: <71d7c64c-0b3a-43bd-8c0f-2c44bf7f78b3@Spark> For testing JSON parsers/encoders I recommend?https://github.com/nst/JSONTestSuite Micha?. On 28 Jun 2018, 10:17 +0200, zxq9@REDACTED, wrote: > On 2018?6?28???? 9?46?10? JST you wrote: > > Hi Craig, > > > > The lib looks interesting, I would like to try it on my projects. > > > > Few related questions: > > 1) Do you plan to add tests later on? > > 2) What do you think about publishing it on hex.pm, so it's easier > > to control releases (for those who are using it), maybe rebar3/erlang.mk > > can be quite helpful here? > > 3) Finally, the performance, did you have a chance to make some comparisons? > > Hi, Oleg! > > 1- I fuzzed it by running real-world JSON through it, but am not convinced > of the utility of hand-written tests considering how narrow the actual > spec is compared to the variety of real-world illegal JSON out there. I > think the best way to *really* approach testing would be to write a script > that would evaluate its performance against either a bulk dataset stored > for that purpose or (much better) use PropEr. > > I'm open to that, but don't have the time right now. It works for my cases > exactly as I intended, but I'm certainly open to contributions. > > I fully realize my position is heretical. > > > 2- If someone wants to push it to hex.pm that would be swell! I don't use > hex.pm in any projects at the moment, so it is a very low priority. Again, > more of a time issue for me than anything. My "extra" time budget wound up > getting spent on making sure the docs came out properly. > > Same as #1, I'm open to that and would love to see it packaged for others > to use in a more simple way -- but it isn't a personal priority for me at > the moment. It will, however, be part of Zomp's default FOSS repository > once I get around to releasing that (so will JSX). > > > 3- The only function I benchmarked formally was decode/1. It performed > about 5% (+/- 10) faster than JSX decode/1 -- so for all practical > purposes is the same. The *size* of the output of decode/1 is larger than > JSX when dealing with strings because decode/1 returns strings-as-strings, > not binaries. This is better for some kinds of manipulation, but isn't > very compact. > > binary_decode/1 almost certainly performs a little slower than decode/1 > because I didn't implement that in a particularly efficient way. I can > improve it to a comparable speed if that becomes necessary, though. > Anyway, this has not been benchmarked formally, just poked at with > timer:tc/1 a few times (but that isn't a particularly reliable way to > benchmark things). > > I have not benchmarked encode/1 or binary_encode/1 at all, but would be > very surprised if they are meaningfully slower than JSX as the approach > taken is very similar -- the mapping between JSON <=> Erlang types being > the important difference. > > > Nice to hear from you! > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Jun 28 10:50:04 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 28 Jun 2018 17:50:04 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <71d7c64c-0b3a-43bd-8c0f-2c44bf7f78b3@Spark> References: <1567611.nGgtL7t0W2@takoyaki> <382545037.n1KB41SD1S@takoyaki> <71d7c64c-0b3a-43bd-8c0f-2c44bf7f78b3@Spark> Message-ID: <29428043.kYQ9HnqbM6@takoyaki> On 2018?6?28???? 10?44?13? JST you wrote: > For testing JSON parsers/encoders I recommend https://github.com/nst/JSONTestSuite Thanks, Micha?! Over the weekend I might hook the two together. If it isn't too complicated I might drop a script for this in there. Very cool. -Craig From gomoripeti@REDACTED Thu Jun 28 12:08:13 2018 From: gomoripeti@REDACTED (=?UTF-8?B?UGV0aSBHw7Ztw7ZyaQ==?=) Date: Thu, 28 Jun 2018 12:08:13 +0200 Subject: [erlang-questions] variable propagation in try Message-ID: Hello, I understand that passing variables between different blocks of a try catch expression is mostly unsafe, because in case of expressions it is not guaranteed that the variable binding actually happened. But why using a variable in a clause after "of" that is bound in the block after "try" is marked as unsafe by the compiler? For example: ``` f() -> try {A, B} = a:b(), A of ok -> B %% unsafe ??? catch _:_ -> a:b() end. ``` thanks Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Jun 28 12:16:42 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 28 Jun 2018 19:16:42 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <29428043.kYQ9HnqbM6@takoyaki> References: <1567611.nGgtL7t0W2@takoyaki> <71d7c64c-0b3a-43bd-8c0f-2c44bf7f78b3@Spark> <29428043.kYQ9HnqbM6@takoyaki> Message-ID: <58854701.HHfBZvMOAq@takoyaki> On 2018?6?28???? 17?50?04? JST zxq9@REDACTED wrote: > On 2018?6?28???? 10?44?13? JST you wrote: > > For testing JSON parsers/encoders I recommend https://github.com/nst/JSONTestSuite > > Thanks, Micha?! > > Over the weekend I might hook the two together. > If it isn't too complicated I might drop a script for this in there. Ran through the test set in the shell just to get some idea, and there are a few cases that do need to be fixed -- all are weird (but legal) or malformed (and some dangerous) number representations. I'll probably put a test function in there that does what I just did in the shell. Including the test set into the repo is a little :-/ Including ZJ into the JSON test project might be a better approach. Once the handful of failing cases are sorted, I don't expect to be patching ZJ much. -Craig From carlsson.richard@REDACTED Thu Jun 28 12:21:49 2018 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 28 Jun 2018 12:21:49 +0200 Subject: [erlang-questions] variable propagation in try In-Reply-To: References: Message-ID: If I recall correctly, it was initially decided to keep the set of safe variables conservative when try/catch was added. It could be relaxed to allow this case. /Richard 2018-06-28 12:08 GMT+02:00 Peti G?m?ri : > Hello, > > I understand that passing variables between different blocks of a try > catch expression is mostly unsafe, because in case of expressions it is not > guaranteed that the variable binding actually happened. > > But why using a variable in a clause after "of" that is bound in the block > after "try" is marked as unsafe by the compiler? > For example: > > ``` > f() -> > try > {A, B} = a:b(), > A > of > ok -> > B %% unsafe ??? > catch _:_ -> > a:b() > end. > ``` > > thanks > Peter > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Jun 28 12:53:18 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 28 Jun 2018 12:53:18 +0200 Subject: [erlang-questions] term_to_binary and large data structures In-Reply-To: References: Message-ID: On Wed, Jun 27, 2018 at 11:24 PM Aaron Seigo wrote: > In fact, for nearly any term you throw at it, this pretty simple > algorithm produces smaller serialized data. You can see the format > employed here: > > https://github.com/aseigo/packer/blob/develop/FORMAT.md > > ?That is also a format with different properties. The external term format doubles as an on-disk format where you might have the need to be robust against a few bad blocks. Schema-based formats tend to be worse than bloaty-prefix-encoded formats here. It probably hurts Elixir more since the map() type is the underlying standard type for many things whereas a record in Erlang is a tuple with some compile time expansion on top. In short, keys are not repeated in this format. You might want to look into Joe Armstrong's UBF stack in which you define the schema as a small stack engine and then interpret that engine to produce data. It serves as a hybrid in which you have a schema, but since the stack engine supports a duplicate-instruction, you can repeat keys in maps if they are the same and so on. In turn, you still have a prefix-like encoding, but it compresses far better for schemes where you have many repeated keys. If you want to have a header-schema, it is probably worth it to just take Protobuf3 and see how well that format handles the data. It has an encoding scheme, varints and ZigZag encoding which represents integers in a way which makes small integers small in the data stream, and also compresses well. So for real-world data, this encoding tend to win. ?Ephemeral data transfer between nodes could benefit from having a new format, or an update which packs maps better. emulator/beam/external.c:BIF_RETTYPE term_to_binary_1(BIF_ALIST_1) ?is the place you want to start looking. Be cautious of the following caveats: * You must write your code so it can be preempted (see the trap variants) * The distribution protocol is a bit different and has an atom-cache for common atoms. I'm not entirely sure this is the entry-point for it ?* We need backwards compatibility. In many cases even small changes to this format has proven to be riddled with loss of compatibility. * We might want to rethink ordering properties of the produced binary. I know this has been a want historically, but I'm not sure we should grant that wish :P * For distribution: More plugability would be really cool to have. Finally, as for the goal of distributing computation: distribute data is still my advice. If data is distributed, computation distributes trivially. Moving data around is going to be a major bottleneck going forward, and the more data you amass, the more you are going to pay moving that data around. Better distribution formats just shaves a constant factor, so you eventually hit the same bottleneck in the long run. The other problem is that any kind of shared/copied data requires locking or coordination. Anything involving those parallelizes badly. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Jun 28 13:15:10 2018 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 28 Jun 2018 13:15:10 +0200 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: References: <1567611.nGgtL7t0W2@takoyaki> <2063693.FJ7v3LWtuf@takoyaki> <0B3CBFEA-A036-44CB-93B2-B90589FE796E@gmail.com> <8a931674-e7be-b827-c5b5-d14679e8b691@ninenines.eu> Message-ID: On Thu, Jun 28, 2018 at 6:50 AM Dmitry Belyaev wrote: > I could never understand where this conversion JSON null <-> Erlang > undefined was coming from. > > Of course we generally don't use null atom in the code, but I always > considered "undefined" value in Erlang meaning "absent" value. For example > when a record is created and some field is not populated explicitly, it is > set undefined. > ?Both variants are somewhat valid: Common Lisp generally treats NIL as the false value, the empty list, null, ... whereas other languages makes the distinction: [], false and null are different things in Erlang for instance. In a statically typed language, you probably would split them. In particular you would have type void type unit = () type json = Int of int | String of string | ... | Null ? ?Where 'void' is the uninhabitable type which can never be realized in a program?, 'unit' is the type of one result (and only that result) and a Null is something of type json. In Erlang, undefined often corresponds to the unit type. It isn't the same as void, because in Erlang every term has to return a value, and you cannot return something of type void. The tradeoff in the end is that of convenience: null == undefined means you can share some data, but you then lost the ability to discriminate null and undefined which from a type/class perspective might be desirable. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Jun 28 14:49:45 2018 From: zxq9@REDACTED (zxq9@REDACTED) Date: Thu, 28 Jun 2018 21:49:45 +0900 Subject: [erlang-questions] New project: ZJ - A tiny JSON encoder/decoder In-Reply-To: <58854701.HHfBZvMOAq@takoyaki> References: <1567611.nGgtL7t0W2@takoyaki> <29428043.kYQ9HnqbM6@takoyaki> <58854701.HHfBZvMOAq@takoyaki> Message-ID: <3137221.VtYOlt1xz4@takoyaki> On 2018?6?28???? 19?16?42? JST zxq9@REDACTED wrote: > On 2018?6?28???? 17?50?04? JST zxq9@REDACTED wrote: > > On 2018?6?28???? 10?44?13? JST you wrote: > > > For testing JSON parsers/encoders I recommend https://github.com/nst/JSONTestSuite > > > > Thanks, Micha?! > > > > Over the weekend I might hook the two together. > > If it isn't too complicated I might drop a script for this in there. > > Ran through the test set in the shell just to get some idea, and there > are a few cases that do need to be fixed -- all are weird (but legal) or > malformed (and some dangerous) number representations. > > I'll probably put a test function in there that does what I just did > in the shell. Including the test set into the repo is a little :-/ > Including ZJ into the JSON test project might be a better approach. > > Once the handful of failing cases are sorted, I don't expect to be > patching ZJ much. All cases that MUST work according to the JSON Test Suite project now do. A handful of cases that SHOULD NOT work do anyway (trailing commas, for example) -- but are mostly of no consequence. A handful of cases that MAY work do, but most do not. Several cases that MAY or SHOULD NOT work return clean or truncated results in decode/1 but return an error in binary_decode/1 due to the way unicode:characters_to_list/1 and unicode:characters_to_binary/1 work. One test case crashes: n_structure_open_array_object.json It is not supposed to work and and crashes many parsers -- this one included. The attempted allocation of tens of thousands of nested maps OOMs the runtime. ZJ does not define (or track) object/array depth. I might implement depth tracking later just for safety. Results are here: https://gitlab.com/zxq9/zj/wikis/json%20test%20suite%20results I don't have time to formalize all this, but I suspect that the JSON definition and test cases won't change much in the future. -Craig From Catenacci@REDACTED Thu Jun 28 15:50:43 2018 From: Catenacci@REDACTED (Onorio Catenacci) Date: Thu, 28 Jun 2018 09:50:43 -0400 Subject: [erlang-questions] [ANN] Chocolatey Nuget Package for OTP 21.0.1 In-Reply-To: References: Message-ID: First, thanks to John H?gberg for making me aware of an issue with Erlang on Windows which is fixed in 21.0.1. I've updated the Chocolatey Nuget package and it's now posted. -- Onorio Catenacci http://onor.io http://www.google.com/+OnorioCatenacci -------------- next part -------------- An HTML attachment was scrubbed... URL: From dane@REDACTED Thu Jun 28 22:35:30 2018 From: dane@REDACTED (Maxim Fedorov) Date: Thu, 28 Jun 2018 20:35:30 +0000 Subject: [erlang-questions] Debugging scheduler not responding to erts_schedule_misc_aux_work Message-ID: <9B6C45BE-65C2-4DB1-8985-D7843ED7AB64@whatsapp.com> I?m trying to debug some weird condition when any misc system task hangs. It seems to affect OTP 20 (but not 16) on FreeBSD 10.3 and 11. It is a rare problem happening after 5-7 days under some load (~40% cpu average on a 48 cores server). There is also a problem with erlang:statistics(runtime), affected by this bug in FreeBSD kernel: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227689 (so statistics:runtime() always returns the same value), however I doubt it can affect anything. What happens: there are several calls, e.g. erlang:statistics(garbage_collection), ets:all(), erts_internal:system_check() and few more. All of them do erts_schedule_misc_aux_work. A misc aux work item is put into every scheduler queue, and it seems that all of them except one respond. VM is still working, all other processes are fine, but the one that did the call is waiting in erlang:gc_info/2 (or another corresponding function), with counter equals to 1. Since there is no timeout in receive statement, it waits forever. How do I debug this? Is there any way to find a scheduler that misbehaves? It is one of the normal schedulers. I?m using gdb to attach to BEAM VM. Unfortunately, I cannot run debug VM (it is not able to handle the load). -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuna.prime@REDACTED Fri Jun 29 00:26:50 2018 From: kuna.prime@REDACTED (Karlo Kuna) Date: Fri, 29 Jun 2018 00:26:50 +0200 Subject: [erlang-questions] custom generators? Message-ID: i was wandering is there protocol or "interface" for making custom generators in erlang? here i asume generator can be used in list comprehensions something similar to I/O protocol -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri Jun 29 01:31:51 2018 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 28 Jun 2018 19:31:51 -0400 Subject: [erlang-questions] custom generators? In-Reply-To: References: Message-ID: <20180628233150.GJ95749@ferdmbp.local> On 06/29, Karlo Kuna wrote: >i was wandering is there protocol or "interface" for making custom >generators in erlang? here i asume generator can be used in list >comprehensions > >something similar to I/O protocol List comprehensions operate only on lists and binaries (using the '<> <= Bin' generator instead of `Pattern <- List'). The closest thing you could get is likely QLC: http://erlang.org/doc/man/qlc.html From kuna.prime@REDACTED Fri Jun 29 01:40:20 2018 From: kuna.prime@REDACTED (Karlo Kuna) Date: Fri, 29 Jun 2018 01:40:20 +0200 Subject: [erlang-questions] custom generators? In-Reply-To: <20180628233150.GJ95749@ferdmbp.local> References: <20180628233150.GJ95749@ferdmbp.local> Message-ID: qlc table seems like answer i was looking for thank you fred On Fri, Jun 29, 2018 at 1:31 AM Fred Hebert wrote: > On 06/29, Karlo Kuna wrote: > >i was wandering is there protocol or "interface" for making custom > >generators in erlang? here i asume generator can be used in list > >comprehensions > > > >something similar to I/O protocol > > List comprehensions operate only on lists and binaries (using the > '<> <= Bin' generator instead of `Pattern <- List'). The > closest thing you could get is likely QLC: > http://erlang.org/doc/man/qlc.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuna.prime@REDACTED Fri Jun 29 01:50:22 2018 From: kuna.prime@REDACTED (Karlo Kuna) Date: Fri, 29 Jun 2018 01:50:22 +0200 Subject: [erlang-questions] when any why to use improper lists Message-ID: dealing with digraph module i have noticed use of improper lists as representations of edges: ['$e' | 123] is there a good reason to use improper lists instead of tuple for this and in general when is a good idea to use improper lists?? (i can't think of example for justified use) -------------- next part -------------- An HTML attachment was scrubbed... URL: From be.dmitry@REDACTED Fri Jun 29 02:23:38 2018 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Fri, 29 Jun 2018 10:23:38 +1000 Subject: [erlang-questions] when any why to use improper lists In-Reply-To: References: Message-ID: It's a way to reduce memory footprint. Tuple of size N is roughly represented in memory as an array [TupleTag, N, TupleElement1, TupleElement2, ..., TupleElementN]. Compare it to Cons cell representation: [ConsTag, HeadElement, TailElement] - it saves 1 word per structure. Kind regards, Dmitry Belyaev On Fri, Jun 29, 2018 at 9:50 AM, Karlo Kuna wrote: > dealing with digraph module i have noticed use of improper lists as > representations of edges: > ['$e' | 123] > > is there a good reason to use improper lists instead of tuple for this and > in general > when is a good idea to use improper lists?? (i can't think of example for > justified use) > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Fri Jun 29 02:35:17 2018 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Fri, 29 Jun 2018 02:35:17 +0200 Subject: [erlang-questions] when any why to use improper lists In-Reply-To: References: Message-ID: A tuple of 2 elements will take 3 words of memory minimum http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html#tuple-arityval-0 tuple1 = [ headerWord, element1, element2 ] A cons cell has no header word, so an improper list of 1 element and 2nd element as a tail, just 2 values stored side to side (same as normal list below except that only 1 cons cell is used, 2 words) cons1 = [ element1, element2 ] A proper list of 2 elements will take 2 cons cells, i.e. 4 words of memory minimum cons2 = [ element2, [] ] cons1 = [ element1, cons2 ] http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html#lists-cons 2018-06-29 2:23 GMT+02:00 Dmitry Belyaev : > It's a way to reduce memory footprint. > > Tuple of size N is roughly represented in memory as an array [TupleTag, N, > TupleElement1, TupleElement2, ..., TupleElementN]. > Compare it to Cons cell representation: [ConsTag, HeadElement, > TailElement] - it saves 1 word per structure. > > Kind regards, > Dmitry Belyaev > > On Fri, Jun 29, 2018 at 9:50 AM, Karlo Kuna wrote: > >> dealing with digraph module i have noticed use of improper lists as >> representations of edges: >> ['$e' | 123] >> >> is there a good reason to use improper lists instead of tuple for this >> and in general >> when is a good idea to use improper lists?? (i can't think of example for >> justified use) >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthonym@REDACTED Fri Jun 29 03:13:08 2018 From: anthonym@REDACTED (anthonym@REDACTED) Date: Thu, 28 Jun 2018 18:13:08 -0700 Subject: [erlang-questions] Debugging scheduler not responding to erts_schedule_misc_aux_work In-Reply-To: <9B6C45BE-65C2-4DB1-8985-D7843ED7AB64@whatsapp.com> References: <9B6C45BE-65C2-4DB1-8985-D7843ED7AB64@whatsapp.com> Message-ID: <1e549a8a2a9f3aa704bba2a1c6e32bff.squirrel@mail.alumni.caltech.edu> It may or may not apply but we had a similar problem with system level work being scheduled on freed processes, and have been debugging it with Ericsson via https://bugs.erlang.org/browse/ERL-573 for the last few months. There's a branch of 20 in one of the later comments which might help, and might be worth a try. HTH, -Anthony > I?m trying to debug some weird condition when any misc system task hangs. > It seems to affect OTP 20 (but not 16) on FreeBSD 10.3 and 11. > > It is a rare problem happening after 5-7 days under some load (~40% cpu > average on a 48 cores server). > > There is also a problem with erlang:statistics(runtime), affected by this > bug in FreeBSD kernel: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227689 (so > statistics:runtime() always returns the same value), however I doubt it > can affect anything. > > What happens: there are several calls, e.g. > erlang:statistics(garbage_collection), ets:all(), > erts_internal:system_check() and few more. All of them do > erts_schedule_misc_aux_work. A misc aux work item is put into every > scheduler queue, and it seems that all of them except one respond. VM is > still working, all other processes are fine, but the one that did the call > is waiting in erlang:gc_info/2 (or another corresponding function), with > counter equals to 1. Since there is no timeout in receive statement, it > waits forever. > > How do I debug this? Is there any way to find a scheduler that misbehaves? > It is one of the normal schedulers. I?m using gdb to attach to BEAM VM. > > Unfortunately, I cannot run debug VM (it is not able to handle the load). > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From dm.klionsky@REDACTED Fri Jun 29 10:16:23 2018 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Fri, 29 Jun 2018 11:16:23 +0300 Subject: [erlang-questions] when any why to use improper lists In-Reply-To: References: Message-ID: <54a1ab03-d318-fd40-e38b-7173df4425b2@gmail.com> Some numbers 1> erts_debug:flat_size([a,b]). 4 2> erts_debug:flat_size({a,b}). 3 3> erts_debug:flat_size([a|b]). 2 On 06/29/2018 03:35 AM, Dmytro Lytovchenko wrote: > A tuple of 2 elements will take 3 words of memory minimum > http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html#tuple-arityval-0 > tuple1 = [ headerWord, element1, element2 ] > > A cons cell has no header word, so an improper list of 1 element and > 2nd element as a tail, just 2 values stored side to side > (same as normal list below except that only 1 cons cell is used, 2 words) > cons1 = [ element1, element2 ] > > A proper list of 2 elements will take 2 cons cells, i.e. 4 words of > memory minimum > cons2 = [ element2, [] ] > cons1 = [ element1, cons2 ] > http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html#lists-cons > > > 2018-06-29 2:23 GMT+02:00 Dmitry Belyaev >: > > It's a way to reduce memory footprint. > > Tuple of size N is roughly represented in memory as an array > [TupleTag, N, TupleElement1, TupleElement2, ..., TupleElementN]. > Compare it to Cons cell representation: [ConsTag, HeadElement, > TailElement] - it saves 1 word per structure. > > Kind regards, > Dmitry Belyaev > > On Fri, Jun 29, 2018 at 9:50 AM, Karlo Kuna > wrote: > > dealing with digraph module i have noticed use of improper > lists as representations of edges: > ['$e' | 123] > > is there a good reason to use improper lists instead of tuple > for this and in general > when is a good idea to use improper lists?? (i can't think of > example for justified use) > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- BR, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-sebastien.pedron@REDACTED Fri Jun 29 12:32:41 2018 From: jean-sebastien.pedron@REDACTED (=?UTF-8?Q?Jean-S=c3=a9bastien_P=c3=a9dron?=) Date: Fri, 29 Jun 2018 12:32:41 +0200 Subject: [erlang-questions] Erlang/OTP 21.0 slow with Windows uniprocessor hosts? Message-ID: Hi! I noticed that Erlang/OTP 21.0.1 was very slow on Windows, when using a single-CPU VM on Amazon EC2 (instance type = t2.micro). Here is the time it takes to run the following one-liner: erl.exe -noinput -eval "halt()." Erlang 20.3: < 1 second (almost immediate) Erlang 21.0.1: 10 seconds I tried with a more powerful instance type (m3.medium) which still provides a single CPU, and the numbers are exactly the same. Now, if the instance type is t2.medium (two CPUs), the times are about the same for both Erlang/OTP 20.3 and 21.0.1 (i.e. < 1 second). I tried to set the number of schedulers to 1 to try to reproduce the situation with I see on t2.micro, but couldn't: both versions of Erlang execute this one-liner quickly. I tried the following more "complex" script: erl.exe -noinput \ -eval "io:format(erlang:system_info(version)), halt()." On t2.micro (one CPU): Erlang/OTP 20.3: < 1 second Erlang/OTP 21.0.1: 20 seconds On t2.medium (two CPUs): Erlang/OTP 20.3: < 1 second Erlang/OTP 21.0.1: < 1 second I also noticed that when Erlang/OTP 21 is slow, it takes 100% CPU. I tried with AMIs of Windows Server 2012 and Windows Server 2016. I suppose this behavior is unexpected. Do you need more details to investigate/reproduce the issue? Thank you! -- Jean-S?bastien P?dron -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From technion@REDACTED Fri Jun 29 14:07:28 2018 From: technion@REDACTED (Technion) Date: Fri, 29 Jun 2018 12:07:28 +0000 Subject: [erlang-questions] eldap hangs Message-ID: Hi, I seem to have a replicable hang in the eldap module. Allow me to start with a working example. If it matters, the LDAP server is a Samba4 AD server. Before anyone asks, these are throw away passwords. Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [hipe] Eshell V10.0 (abort with ^G) 1> ===> The rebar3 shell is a development tool; to deploy applications in production, consider using releases (http://www.rebar3.org/docs/releases) ===> Booted p3 ===> Booted sasl 1> {ok, H} = eldap:open(["127.0.0.1"], [{port, 1636}, {ssl, true}, {timeout, 500}]). {ok,<0.131.0>} 2> ok = eldap:simple_bind(H, "Kevin@REDACTED", "Kevin111"). ok So we definitely have of using this. Now let's see what happens in a new session if we first get the username wrong, then try it correctly: 1> {ok, H} = eldap:open(["127.0.0.1"], [{port, 1636}, {ssl, true}, {timeout, 500}]). {ok,<0.131.0>} 2> ok = eldap:simple_bind(H, "Kevin", "Kevin111"). ** exception error: no match of right hand side value {error,invalidCredentials} 3> ok = eldap:simple_bind(H, "Kevin@REDACTED", "Kevin111"). Given I'm trying to build an interface where users can enter their own username, well it creates an easily hung session. The docs describe a timeout on open/2 but it obviously doesn't apply to everything. Given I"m not seeing any output at all I have no real idea where to debug this. Any assistance appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Fri Jun 29 14:14:22 2018 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Fri, 29 Jun 2018 15:14:22 +0300 Subject: [erlang-questions] eldap hangs In-Reply-To: References: Message-ID: <0d3a8f3a-7fb1-ca94-b962-d14af3ed85ca@gmail.com> The wrong password call raised an exception and the process behind H (<0.131.0> is dead. And I think this is the reason of hang. You can start your session with > catch_exception(true). This is catch exceptions and your H will survive the crash and should work. On 06/29/2018 03:07 PM, Technion wrote: > > Hi, > > > I seem to have a replicable hang in the?eldap module. Allow me to > start with a working example. If it matters, the LDAP server is a > Samba4 AD server. Before anyone asks, these are throw away passwords. > > > Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:1:1] [ds:1:1:10] > [async-threads:1] [hipe] > > Eshell V10.0? (abort with ^G) > 1> ===> The rebar3 shell is a development tool; to deploy applications > in production, consider using releases > (http://www.rebar3.org/docs/releases) > ===> Booted p3 > ===> Booted sasl > > 1> {ok, H} = eldap:open(["127.0.0.1"], [{port, 1636}, {ssl, true}, > {timeout, 500}]). > {ok,<0.131.0>} > 2> ok = eldap:simple_bind(H, "Kevin@REDACTED", "Kevin111"). > ok > > So we definitely have of using this. Now let's see what happens in a > new session?if we first get the username wrong, then try it correctly: > > 1> {ok, H} = eldap:open(["127.0.0.1"], [{port, 1636}, {ssl, true}, > {timeout, 500}]). > {ok,<0.131.0>} > 2>? ok = eldap:simple_bind(H, "Kevin", "Kevin111"). > ** exception error: no match of right hand side value > {error,invalidCredentials} > 3> ok = eldap:simple_bind(H, "Kevin@REDACTED", "Kevin111"). > > > Given I'm trying to build an interface where users can enter their own > username, well it creates an easily hung session. The docs describe a > timeout on open/2 but it obviously doesn't apply to everything. Given > I"m not seeing any output at all I have no real idea where to debug this. > > > Any assistance appreciated. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- BR, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Fri Jun 29 14:14:13 2018 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Fri, 29 Jun 2018 14:14:13 +0200 Subject: [erlang-questions] when any why to use improper lists In-Reply-To: <54a1ab03-d318-fd40-e38b-7173df4425b2@gmail.com> References: <54a1ab03-d318-fd40-e38b-7173df4425b2@gmail.com> Message-ID: Has there been some testing / measurements done on the memory saved by introducing a "tuple of 2" tag only for 2-tuples? {ok, Bla} or {error, R} is such a common pattern and "most" machines being 64bits nowadays this should save many times 8 bytes, shouldn't it? So we have a tag for lists (cons), a tag for n-sized tuples and tags for other data. How about a tag for a memory structure that's basically [a|b] but which would be understood as {a,b}. Cheers, -- Pierre Fenoll On Fri, 29 Jun 2018 at 10:16, Dmitry Klionsky wrote: > Some numbers > > 1> erts_debug:flat_size([a,b]). > 4 > 2> erts_debug:flat_size({a,b}). > 3 > 3> erts_debug:flat_size([a|b]). > 2 > > > On 06/29/2018 03:35 AM, Dmytro Lytovchenko wrote: > > A tuple of 2 elements will take 3 words of memory minimum > > http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html#tuple-arityval-0 > tuple1 = [ headerWord, element1, element2 ] > > A cons cell has no header word, so an improper list of 1 element and 2nd > element as a tail, just 2 values stored side to side > (same as normal list below except that only 1 cons cell is used, 2 words) > cons1 = [ element1, element2 ] > > A proper list of 2 elements will take 2 cons cells, i.e. 4 words of memory > minimum > cons2 = [ element2, [] ] > cons1 = [ element1, cons2 ] > http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html#lists-cons > > > 2018-06-29 2:23 GMT+02:00 Dmitry Belyaev : > >> It's a way to reduce memory footprint. >> >> Tuple of size N is roughly represented in memory as an array [TupleTag, >> N, TupleElement1, TupleElement2, ..., TupleElementN]. >> Compare it to Cons cell representation: [ConsTag, HeadElement, >> TailElement] - it saves 1 word per structure. >> >> Kind regards, >> Dmitry Belyaev >> >> On Fri, Jun 29, 2018 at 9:50 AM, Karlo Kuna wrote: >> >>> dealing with digraph module i have noticed use of improper lists as >>> representations of edges: >>> ['$e' | 123] >>> >>> is there a good reason to use improper lists instead of tuple for this >>> and in general >>> when is a good idea to use improper lists?? (i can't think of example >>> for justified use) >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -- > BR, > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Fri Jun 29 14:30:21 2018 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Fri, 29 Jun 2018 14:30:21 +0200 Subject: [erlang-questions] when any why to use improper lists In-Reply-To: References: <54a1ab03-d318-fd40-e38b-7173df4425b2@gmail.com> Message-ID: A problem, cons is a pointer. Tuple is also a pointer. Currently 2 bits are used for such tags, and adding more special cases will need more bits. Tagged value must have enough bits to hold a whole pointer, so by using the fact that all memory words are 4 (8) byte-aligned, there are only 2 bits available (for compatibility with 32 bit systems) and on 64-bit it could become 3 bit to fit some extra special cases in there. This 3-bit tagging is not implemented in BEAM VM. With existing C code in BEAM VM i am sure it will create a lot of suffering for a person trying to implement it. Another trick to tag a pointer is to use high bits above 48 or 56 which depending on platform may be unused and on Intel must always be set to 1. This trick is not portable and is not used. 2018-06-29 14:14 GMT+02:00 Pierre Fenoll : > Has there been some testing / measurements done on the memory saved by > introducing a "tuple of 2" tag only for 2-tuples? > {ok, Bla} or {error, R} is such a common pattern and "most" machines being > 64bits nowadays this should save many times 8 bytes, shouldn't it? > So we have a tag for lists (cons), a tag for n-sized tuples and tags for > other data. How about a tag for a memory structure that's basically [a|b] > but which would be understood as {a,b}. > > Cheers, > -- > Pierre Fenoll > > > > On Fri, 29 Jun 2018 at 10:16, Dmitry Klionsky > wrote: > >> Some numbers >> >> 1> erts_debug:flat_size([a,b]). >> 4 >> 2> erts_debug:flat_size({a,b}). >> 3 >> 3> erts_debug:flat_size([a|b]). >> 2 >> >> >> On 06/29/2018 03:35 AM, Dmytro Lytovchenko wrote: >> >> A tuple of 2 elements will take 3 words of memory minimum >> http://beam-wisdoms.clau.se/en/latest/indepth-memory- >> layout.html#tuple-arityval-0 >> tuple1 = [ headerWord, element1, element2 ] >> >> A cons cell has no header word, so an improper list of 1 element and 2nd >> element as a tail, just 2 values stored side to side >> (same as normal list below except that only 1 cons cell is used, 2 words) >> cons1 = [ element1, element2 ] >> >> A proper list of 2 elements will take 2 cons cells, i.e. 4 words of >> memory minimum >> cons2 = [ element2, [] ] >> cons1 = [ element1, cons2 ] >> http://beam-wisdoms.clau.se/en/latest/indepth-memory- >> layout.html#lists-cons >> >> >> 2018-06-29 2:23 GMT+02:00 Dmitry Belyaev : >> >>> It's a way to reduce memory footprint. >>> >>> Tuple of size N is roughly represented in memory as an array [TupleTag, >>> N, TupleElement1, TupleElement2, ..., TupleElementN]. >>> Compare it to Cons cell representation: [ConsTag, HeadElement, >>> TailElement] - it saves 1 word per structure. >>> >>> Kind regards, >>> Dmitry Belyaev >>> >>> On Fri, Jun 29, 2018 at 9:50 AM, Karlo Kuna >>> wrote: >>> >>>> dealing with digraph module i have noticed use of improper lists as >>>> representations of edges: >>>> ['$e' | 123] >>>> >>>> is there a good reason to use improper lists instead of tuple for this >>>> and in general >>>> when is a good idea to use improper lists?? (i can't think of example >>>> for justified use) >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> >> >> _______________________________________________ >> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions >> >> >> -- >> BR, >> Dmitry >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From technion@REDACTED Fri Jun 29 14:48:05 2018 From: technion@REDACTED (Technion) Date: Fri, 29 Jun 2018 12:48:05 +0000 Subject: [erlang-questions] eldap hangs In-Reply-To: <0d3a8f3a-7fb1-ca94-b962-d14af3ed85ca@gmail.com> References: , <0d3a8f3a-7fb1-ca94-b962-d14af3ed85ca@gmail.com> Message-ID: Hi Dmitry, Thanks a heap, you are correct. The process does disappear after the incorrect password. 10> process_info(H). undefined ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Dmitry Klionsky Sent: Friday, 29 June 2018 10:14 PM To: erlang-questions@REDACTED Subject: Re: [erlang-questions] eldap hangs The wrong password call raised an exception and the process behind H (<0.131.0> is dead. And I think this is the reason of hang. You can start your session with > catch_exception(true). This is catch exceptions and your H will survive the crash and should work. On 06/29/2018 03:07 PM, Technion wrote: Hi, I seem to have a replicable hang in the eldap module. Allow me to start with a working example. If it matters, the LDAP server is a Samba4 AD server. Before anyone asks, these are throw away passwords. Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [hipe] Eshell V10.0 (abort with ^G) 1> ===> The rebar3 shell is a development tool; to deploy applications in production, consider using releases (http://www.rebar3.org/docs/releases) ===> Booted p3 ===> Booted sasl 1> {ok, H} = eldap:open(["127.0.0.1"], [{port, 1636}, {ssl, true}, {timeout, 500}]). {ok,<0.131.0>} 2> ok = eldap:simple_bind(H, "Kevin@REDACTED", "Kevin111"). ok So we definitely have of using this. Now let's see what happens in a new session if we first get the username wrong, then try it correctly: 1> {ok, H} = eldap:open(["127.0.0.1"], [{port, 1636}, {ssl, true}, {timeout, 500}]). {ok,<0.131.0>} 2> ok = eldap:simple_bind(H, "Kevin", "Kevin111"). ** exception error: no match of right hand side value {error,invalidCredentials} 3> ok = eldap:simple_bind(H, "Kevin@REDACTED", "Kevin111"). Given I'm trying to build an interface where users can enter their own username, well it creates an easily hung session. The docs describe a timeout on open/2 but it obviously doesn't apply to everything. Given I"m not seeing any output at all I have no real idea where to debug this. Any assistance appreciated. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -- BR, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From dieter@REDACTED Fri Jun 29 12:41:36 2018 From: dieter@REDACTED (dieter@REDACTED) Date: Fri, 29 Jun 2018 10:41:36 +0000 Subject: [erlang-questions] custom generators? In-Reply-To: References: Message-ID: Hi Karlo, list, I am just learning erlang and found this an interesting topic to explore. I wrote a little server, which creates generator functions. Right now it is not usable for list comprehensions, I think. And the termination is not very nice yet.. But maybe someone finds it useful. Kind regards, dieter ------------------------------------------------------------------------ -module(generator). -behaviour(gen_server). %% API -export([generate/2]). %% gen_server callbacks -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). -define(SERVER, ?MODULE). -record(state, {client_state, client_fun}). %%%=================================================================== %%% API %%%=================================================================== generate(Function,State) -> {ok, Pid} = gen_server:start_link(?MODULE, [Function,State], []), Generator = fun(Params) -> {ok, Value} = gen_server:call(Pid, {next, Params}), Value end, Generator. init([Function, State]) -> {ok, #state{client_fun=Function, client_state=State}}. handle_call({next, Params}, _From, State = #state{client_fun=Function, client_state=ClientState}) -> case Function(Params, ClientState) of {ok, Value, NewClientState} -> {reply, {ok, Value}, State#state{client_state=NewClientState}}; stop -> {stop, stopped, State} end. [..] %% Client callback function definition %% @spec func(Params, State) -> {ok, Value, State} | stop %% Test run 1> G = generator:generate(fun(P,S) -> V=P+S, NS=S+1, case NS of 13 -> stop; _ -> {ok, V, NS} end end, 10). #Fun 2> 2> 2> 2> G13(1). 11 3> G13(1). 12 4> G13(1). ** exception exit: stopped 5> =ERROR REPORT==== 29-Jun-2018::10:32:54 === ** Generic server terminating ** Last message in was {next,1} ** When Server state == {state,12,#Fun} ** Reason for termination == ** stopped ** Client stacktrace ** [{gen,do_call,4,[{file,"gen.erl"},{line,169}]}, {gen_server,call,2,[{file,"gen_server.erl"},{line,202}]}, {generator,'-generate/2-fun-0-',2,[{file,"generator.erl"},{line,30}]}, {erl_eval,do_apply,5,[{file,"erl_eval.erl"},{line,661}]}, {shell,exprs,7,[{file,"shell.erl"},{line,687}]}, {shell,eval_exprs,7,[{file,"shell.erl"},{line,642}]}, {shell,eval_loop,3,[{file,"shell.erl"},{line,627}]}] Am Fr., Jun. 29, 2018 00:27 schrieb Karlo Kuna : i was wandering is there protocol or "interface" for making custom generators in erlang? here i asume generator can be used in list comprehensions something similar to I/O protocol -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker@REDACTED Fri Jun 29 15:30:07 2018 From: sverker@REDACTED (Sverker Eriksson) Date: Fri, 29 Jun 2018 15:30:07 +0200 Subject: [erlang-questions] Patch package OTP 20.3.8.2 released Message-ID: <1530279007.28175.4.camel@erlang.org> Patch Package:???????????OTP 20.3.8.2 Git Tag:?????????????????OTP-20.3.8.2 Date:????????????????????2018-06-29 Trouble Report Id:???????OTP-15067, OTP-15160, OTP-15161, OTP-15162 Seq num:?????????????????ERIERL-191, ERL-573 System:??????????????????OTP Release:?????????????????20 Application:?????????????erl_interface-3.10.2.1, erts-9.3.3.1, ?????????????????????????ic-4.4.4.1, kernel-5.4.3.1 Predecessor:?????????????OTP 20.3.8.1 ?Check out the git tag OTP-20.3.8.2, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- erl_interface-3.10.2.1 ------------------------------------------ ?--------------------------------------------------------------------- ?The erl_interface-3.10.2.1 application can be applied independently ?of other applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15161????Application(s): erl_interface ???????????????Related Id(s): ERIERL-191 ???????????????Make ei_connect and friends also accept state ???????????????ok_simultaneous during handshake, which means the other ???????????????node has initiated a connection setup that will be ???????????????cancelled in favor of this connection. ?--------------------------------------------------------------------- ?--- erts-9.3.3.1 ---------------------------------------------------- ?--------------------------------------------------------------------- ?The erts-9.3.3.1 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15067????Application(s): erts ???????????????Related Id(s): ERL-573 ???????????????Fixed a rare bug that could cause processes to be ???????????????scheduled after they had been freed. ?Full runtime dependencies of erts-9.3.3.1: kernel-5.0, sasl-3.0.1, ?stdlib-3.0 ?--------------------------------------------------------------------- ?--- ic-4.4.4.1 ------------------------------------------------------ ?--------------------------------------------------------------------- ?The ic-4.4.4.1 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-15160????Application(s): ic ???????????????Related Id(s): ERIERL-191 ???????????????Fixed bug in ic causing potential buffer overrun in ???????????????funtion oe_ei_encode_atom. Bug exists since ic-4.4.4 ???????????????(OTP-20.3.4). ?Full runtime dependencies of ic-4.4.4.1: erts-6.0, kernel-3.0, ?stdlib-2.0 ?--------------------------------------------------------------------- ?--- kernel-5.4.3.1 -------------------------------------------------- ?--------------------------------------------------------------------- ?Note! The kernel-5.4.3.1 application can *not* be applied ???????independently of other applications on an arbitrary OTP 20 ???????installation. ???????On a full OTP 20 installation, also the following runtime ???????dependency has to be satisfied: ???????-- erts-9.3 (first satisfied in OTP 20.3) ?--- Fixed Bugs and Malfunctions --- ? OTP-15162????Application(s): kernel ???????????????Related Id(s): ERIERL-191 ???????????????Fix some potential buggy behavior in how ticks are sent ???????????????on inter node distribution connections. Tick is now ???????????????sent to c-node even if there are unsent buffered data, ???????????????as c-nodes need ticks in order to send reply ticks. The ???????????????amount of sent data was also calculated wrongly when ???????????????ticks were suppressed due to unsent buffered data. ?Full runtime dependencies of kernel-5.4.3.1: erts-9.3, sasl-3.0, ?stdlib-3.4 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- From anthonym@REDACTED Fri Jun 29 18:46:31 2018 From: anthonym@REDACTED (Anthony Molinaro) Date: Fri, 29 Jun 2018 09:46:31 -0700 Subject: [erlang-questions] Debugging scheduler not responding to erts_schedule_misc_aux_work In-Reply-To: <1e549a8a2a9f3aa704bba2a1c6e32bff.squirrel@mail.alumni.caltech.edu> References: <9B6C45BE-65C2-4DB1-8985-D7843ED7AB64@whatsapp.com> <1e549a8a2a9f3aa704bba2a1c6e32bff.squirrel@mail.alumni.caltech.edu> Message-ID: And it looks like this got merged into 20.3.8.2 released today, so you can try that tag to see if it fixes your problem. HTH, -Anthony > On Jun 28, 2018, at 6:13 PM, anthonym@REDACTED wrote: > > It may or may not apply but we had a similar problem with system level > work being scheduled on freed processes, and have been debugging it with > Ericsson via > > https://bugs.erlang.org/browse/ERL-573 > > for the last few months. There's a branch of 20 in one of the later > comments which might help, and might be worth a try. > > HTH, > > -Anthony > > >> I?m trying to debug some weird condition when any misc system task hangs. >> It seems to affect OTP 20 (but not 16) on FreeBSD 10.3 and 11. >> >> It is a rare problem happening after 5-7 days under some load (~40% cpu >> average on a 48 cores server). >> >> There is also a problem with erlang:statistics(runtime), affected by this >> bug in FreeBSD kernel: >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227689 (so >> statistics:runtime() always returns the same value), however I doubt it >> can affect anything. >> >> What happens: there are several calls, e.g. >> erlang:statistics(garbage_collection), ets:all(), >> erts_internal:system_check() and few more. All of them do >> erts_schedule_misc_aux_work. A misc aux work item is put into every >> scheduler queue, and it seems that all of them except one respond. VM is >> still working, all other processes are fine, but the one that did the call >> is waiting in erlang:gc_info/2 (or another corresponding function), with >> counter equals to 1. Since there is no timeout in receive statement, it >> waits forever. >> >> How do I debug this? Is there any way to find a scheduler that misbehaves? >> It is one of the normal schedulers. I?m using gdb to attach to BEAM VM. >> >> Unfortunately, I cannot run debug VM (it is not able to handle the load). >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mikpelinux@REDACTED Fri Jun 29 19:10:29 2018 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Fri, 29 Jun 2018 19:10:29 +0200 Subject: [erlang-questions] when any why to use improper lists In-Reply-To: References: <54a1ab03-d318-fd40-e38b-7173df4425b2@gmail.com> Message-ID: With the current tag scheme, special-casing 2-element tuples would not be a win space-wise, as we cannot make them header-less without sacrificing something else, and that's not going to happen. With 3 primary tag bits we could achieve that, but then general tuple tests would be more complicated since multiple primary tags would map to tuples. However, OTP should definitely go to a 64-bit, 3 primary tag bits, tag scheme soon. Who cares about 32-bit machines, right? Just use CONS cells for 2-tuples of there's no confusion and you're behind an ADT layer. On Fri, Jun 29, 2018 at 2:30 PM, Dmytro Lytovchenko wrote: > A problem, cons is a pointer. Tuple is also a pointer. Currently 2 bits are > used for such tags, and adding more special cases will need more bits. > > Tagged value must have enough bits to hold a whole pointer, so by using the > fact that all memory words are 4 (8) byte-aligned, there are only 2 bits > available (for compatibility with 32 bit systems) and on 64-bit it could > become 3 bit to fit some extra special cases in there. This 3-bit tagging is > not implemented in BEAM VM. With existing C code in BEAM VM i am sure it > will create a lot of suffering for a person trying to implement it. > > Another trick to tag a pointer is to use high bits above 48 or 56 which > depending on platform may be unused and on Intel must always be set to 1. > This trick is not portable and is not used. > > 2018-06-29 14:14 GMT+02:00 Pierre Fenoll : >> >> Has there been some testing / measurements done on the memory saved by >> introducing a "tuple of 2" tag only for 2-tuples? >> {ok, Bla} or {error, R} is such a common pattern and "most" machines being >> 64bits nowadays this should save many times 8 bytes, shouldn't it? >> So we have a tag for lists (cons), a tag for n-sized tuples and tags for >> other data. How about a tag for a memory structure that's basically [a|b] >> but which would be understood as {a,b}. >> >> Cheers, >> -- >> Pierre Fenoll >> >> >> >> On Fri, 29 Jun 2018 at 10:16, Dmitry Klionsky >> wrote: >>> >>> Some numbers >>> >>> 1> erts_debug:flat_size([a,b]). >>> 4 >>> 2> erts_debug:flat_size({a,b}). >>> 3 >>> 3> erts_debug:flat_size([a|b]). >>> 2 >>> >>> >>> On 06/29/2018 03:35 AM, Dmytro Lytovchenko wrote: >>> >>> A tuple of 2 elements will take 3 words of memory minimum >>> >>> http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html#tuple-arityval-0 >>> tuple1 = [ headerWord, element1, element2 ] >>> >>> A cons cell has no header word, so an improper list of 1 element and 2nd >>> element as a tail, just 2 values stored side to side >>> (same as normal list below except that only 1 cons cell is used, 2 words) >>> cons1 = [ element1, element2 ] >>> >>> A proper list of 2 elements will take 2 cons cells, i.e. 4 words of >>> memory minimum >>> cons2 = [ element2, [] ] >>> cons1 = [ element1, cons2 ] >>> >>> http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html#lists-cons >>> >>> >>> 2018-06-29 2:23 GMT+02:00 Dmitry Belyaev : >>>> >>>> It's a way to reduce memory footprint. >>>> >>>> Tuple of size N is roughly represented in memory as an array [TupleTag, >>>> N, TupleElement1, TupleElement2, ..., TupleElementN]. >>>> Compare it to Cons cell representation: [ConsTag, HeadElement, >>>> TailElement] - it saves 1 word per structure. >>>> >>>> Kind regards, >>>> Dmitry Belyaev >>>> >>>> On Fri, Jun 29, 2018 at 9:50 AM, Karlo Kuna >>>> wrote: >>>>> >>>>> dealing with digraph module i have noticed use of improper lists as >>>>> representations of edges: >>>>> ['$e' | 123] >>>>> >>>>> is there a good reason to use improper lists instead of tuple for this >>>>> and in general >>>>> when is a good idea to use improper lists?? (i can't think of example >>>>> for justified use) >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> -- >>> BR, >>> Dmitry >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From dch@REDACTED Sat Jun 30 10:12:26 2018 From: dch@REDACTED (Dave Cottlehuber) Date: Sat, 30 Jun 2018 10:12:26 +0200 Subject: [erlang-questions] Is there a way to disable EPMD in Erlang releases? In-Reply-To: <2043113197.3712104.1530152721219@mail.yahoo.com> References: <2043113197.3712104.1530152721219.ref@mail.yahoo.com> <2043113197.3712104.1530152721219@mail.yahoo.com> Message-ID: <1530346346.1055785.1425497272.529F3A78@webmail.messagingengine.com> On Thu, 28 Jun 2018, at 04:25, Vans S wrote: > Hello, > I am currently compiling an Erlang release of a PoC app that is > ending up being used on Win Mac and Linux. Is there any way to > tell the> release to not include and/or not start EPMD? Add this this to your vm.args -start_epmd false More details in http://erlang.org/doc/man/erl.html dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From dch@REDACTED Sat Jun 30 14:06:20 2018 From: dch@REDACTED (Dave Cottlehuber) Date: Sat, 30 Jun 2018 14:06:20 +0200 Subject: [erlang-questions] Debugging scheduler not responding to erts_schedule_misc_aux_work In-Reply-To: <9B6C45BE-65C2-4DB1-8985-D7843ED7AB64@whatsapp.com> References: <9B6C45BE-65C2-4DB1-8985-D7843ED7AB64@whatsapp.com> Message-ID: <1530360380.1127238.1425601176.7EC4B0D9@webmail.messagingengine.com> On Thu, 28 Jun 2018, at 22:35, Maxim Fedorov wrote: > I?m trying to debug some weird condition when any misc system > task hangs.> It seems to affect OTP 20 (but not 16) on FreeBSD 10.3 and 11. > > It is a rare problem happening after 5-7 days under some load > (~40% cpu> average on a 48 cores server). > > There is also a problem with erlang:statistics(runtime), affected by > this bug in FreeBSD kernel: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227689 (so > statistics:runtime() always returns the same value), however I > doubt it> can affect anything. > > What happens: there are several calls, e.g. > erlang:statistics(garbage_collection), ets:all(), > erts_internal:system_check() and few more. All of them do > erts_schedule_misc_aux_work. A misc aux work item is put into every > scheduler queue, and it seems that all of them except one > respond. VM is> still working, all other processes are fine, but the one that did the> call is waiting in erlang:gc_info/2 (or another corresponding > function),> with counter equals to 1. Since there is no timeout in receive > statement, it waits forever. > > How do I debug this? I get that your load in production is high, but will a targeted dtrace probe be lightweight enough? Also wrt Anthony?s comments are you building with clang or gcc? I?m not clear if that?s relevant A+ Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: