From eric.des.courtis@REDACTED Wed Nov 1 00:05:47 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Tue, 31 Oct 2017 19:05:47 -0400 Subject: [erlang-questions] How does Erlang TCP determine the end of a TCP stream? In-Reply-To: References: <3A3B337C-E459-4039-BD4A-8805EFD7DC07@gmail.com> Message-ID: On Tue, Oct 31, 2017 at 4:04 PM, scott ribe wrote: > On Oct 31, 2017, at 1:38 PM, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED> wrote: > > > > If you can, run with something like {packet, 4}. It is simple and works. > Delimiters are worse because you have to scan for them and you often need > to escape them in your payload. A Length-Value-Type encoding provides > framing in which you don't have to scan and you don't need to escape. > Putting the type last in the packet is debatable, but one advantage is that > other languages have to allocate a full buffer which can often eliminate > certain security-concerning bad implementations. > > I like length-prefixed messages, and I like that Erlang has such direct & > easy support built-in. I'm just going to suggest one more thing: add your > own checksum at the end. In case there's any bug anywhere in decoding > packets, this prevents the kind of error where you get out of sync and are > then taking what should be message data as your 4 bytes of length. > I think this is probably unnecessary since an implementation for something like {packet,4} is trivial to begin with so you will likely never see an issue with it. Not to mention UDP and TCP both have checksums checked by the OS anyway. > > Admittedly, that is far less likely with OTP's built-in implementation > than if you're writing your buffering and decoding. But it's also possible > for a sender to send a bogus length, or maybe the length is right but it > fails to send the last byte before sending the length of the next message... > > -- > Scott Ribe > https://www.linkedin.com/in/scottribe/ > (303) 722-0567 > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott_ribe@REDACTED Wed Nov 1 00:31:46 2017 From: scott_ribe@REDACTED (scott ribe) Date: Tue, 31 Oct 2017 17:31:46 -0600 Subject: [erlang-questions] How does Erlang TCP determine the end of a TCP stream? In-Reply-To: References: <3A3B337C-E459-4039-BD4A-8805EFD7DC07@gmail.com> Message-ID: <0F2A7123-456F-46E8-9724-95AC81505A55@elevated-dev.com> On Oct 31, 2017, at 5:05 PM, Eric des Courtis wrote: > > I think this is probably unnecessary since an implementation for something like {packet,4} is trivial to begin with so you will likely never see an issue with it. Not to mention UDP and TCP both have checksums checked by the OS anyway. As I pointed out, you can have a bug on the sending end as well. And the TCP checksum is extremely weak, basically just sum of words--that's fine until you have a terribly noisy corrupt transport... So yes, in a lot of circumstances it's overkill, but I prefer to be paranoid when a network is involved. -- Scott Ribe https://www.linkedin.com/in/scottribe/ (303) 722-0567 From eric.des.courtis@REDACTED Wed Nov 1 01:09:18 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Tue, 31 Oct 2017 20:09:18 -0400 Subject: [erlang-questions] How does Erlang TCP determine the end of a TCP stream? In-Reply-To: <0F2A7123-456F-46E8-9724-95AC81505A55@elevated-dev.com> References: <3A3B337C-E459-4039-BD4A-8805EFD7DC07@gmail.com> <0F2A7123-456F-46E8-9724-95AC81505A55@elevated-dev.com> Message-ID: Depends on the data and what you are doing with it. If it's say a video over a gigantic P2P network then yes I would agree but then you might want something like SHA256 or SHA3 as your checksum. On Tue, Oct 31, 2017 at 7:31 PM, scott ribe wrote: > On Oct 31, 2017, at 5:05 PM, Eric des Courtis > wrote: > > > > I think this is probably unnecessary since an implementation for > something like {packet,4} is trivial to begin with so you will likely never > see an issue with it. Not to mention UDP and TCP both have checksums > checked by the OS anyway. > > As I pointed out, you can have a bug on the sending end as well. And the > TCP checksum is extremely weak, basically just sum of words--that's fine > until you have a terribly noisy corrupt transport... > > So yes, in a lot of circumstances it's overkill, but I prefer to be > paranoid when a network is involved. > > -- > Scott Ribe > https://www.linkedin.com/in/scottribe/ > (303) 722-0567 > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.des.courtis@REDACTED Wed Nov 1 01:11:44 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Tue, 31 Oct 2017 20:11:44 -0400 Subject: [erlang-questions] How does Erlang TCP determine the end of a TCP stream? In-Reply-To: References: <3A3B337C-E459-4039-BD4A-8805EFD7DC07@gmail.com> <0F2A7123-456F-46E8-9724-95AC81505A55@elevated-dev.com> Message-ID: Then again that checksum should probably stick around with the data because hard drives, CPUs and RAM are more likely to screw up the data. On Tue, Oct 31, 2017 at 8:09 PM, Eric des Courtis < eric.des.courtis@REDACTED> wrote: > Depends on the data and what you are doing with it. If it's say a video > over a gigantic P2P network then yes I would agree but then you might want > something like SHA256 or SHA3 as your checksum. > > On Tue, Oct 31, 2017 at 7:31 PM, scott ribe > wrote: > >> On Oct 31, 2017, at 5:05 PM, Eric des Courtis >> wrote: >> > >> > I think this is probably unnecessary since an implementation for >> something like {packet,4} is trivial to begin with so you will likely never >> see an issue with it. Not to mention UDP and TCP both have checksums >> checked by the OS anyway. >> >> As I pointed out, you can have a bug on the sending end as well. And the >> TCP checksum is extremely weak, basically just sum of words--that's fine >> until you have a terribly noisy corrupt transport... >> >> So yes, in a lot of circumstances it's overkill, but I prefer to be >> paranoid when a network is involved. >> >> -- >> Scott Ribe >> https://www.linkedin.com/in/scottribe/ >> (303) 722-0567 >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott_ribe@REDACTED Wed Nov 1 05:05:35 2017 From: scott_ribe@REDACTED (scott ribe) Date: Tue, 31 Oct 2017 22:05:35 -0600 Subject: [erlang-questions] How does Erlang TCP determine the end of a TCP stream? In-Reply-To: References: <3A3B337C-E459-4039-BD4A-8805EFD7DC07@gmail.com> <0F2A7123-456F-46E8-9724-95AC81505A55@elevated-dev.com> Message-ID: <314CF683-14E4-445B-BC1F-DD4916F77B78@elevated-dev.com> On Oct 31, 2017, at 6:09 PM, Eric des Courtis wrote: > > Depends on the data and what you are doing with it. If it's say a video over a gigantic P2P network then yes I would agree but then you might want something like SHA256 or SHA3 as your checksum. Exactly. I'd probably do murmur or SIP hash for performance, but that's basically the idea. > Then again that checksum should probably stick around with the data because hard drives, CPUs and RAM are more likely to screw up the data. I'm a fan of ZFS ;-) -- Scott Ribe https://www.linkedin.com/in/scottribe/ (303) 722-0567 From askjuise@REDACTED Wed Nov 1 10:10:18 2017 From: askjuise@REDACTED (Alexander Petrovsky) Date: Wed, 1 Nov 2017 12:10:18 +0300 Subject: [erlang-questions] run_erl and systemd In-Reply-To: References: <1509433072.2295.97.camel@fmi.fi> Message-ID: Hi! Looks like you observe very strange behaviour, we run our erlang release in OpenStack ubuntu instance, with follow systemd config and everything works fine: [Unit] After=network.target remote-fs.target Requires=network.target remote-fs.target [Service] Type=forking TimeoutStartSec=0 Restart=always RestartSec=5 Environment=HOME=/root ExecStart={SCRIPT} start ExecStop={SCRIPT} stop [Install] WantedBy=multi-user.target Note that run_erl complain when HOME var is not defined. Also, I suggest try to get rid of User and Group. 2017-10-31 12:52 GMT+03:00 Ali Sabil : > I tend to use `/proj/myapp/bin/myapp foreground` and avoid run_erl > altogether. > > On Tue, Oct 31, 2017 at 12:27 PM, Ryn? Jouni (FMI) > wrote: > >> Hi list >> >> I have an odd systemd and run_erl issue. This service definition more >> or less "works" on a "physical" machine, but not on a KVM-virtual >> machine. >> >> [Unit] >> Description=My node >> After=network.target >> >> [Service] >> Environment=RUNNER_LOG_DIR=/var/log/myapp >> Type=forking >> ExecStartPre=/bin/sleep 60 >> ExecStart=/proj/myapp/bin/myapp start >> ExecStop=/proj/myapp/bin/myapp stop >> RestartSec=20 >> Restart=on-failure >> User=me >> Group=me >> >> [Install] >> WantedBy=multi-user.target >> >> >> epmd is running and I just try to start it via "systemctl start >> my.service" from a command line >> >> Plus I do get some odd inet_gethost and epmd died errors even in the >> physical machine, but at least the run_erl starts at some point ... >> >> >> Is there something, that I should do differently or add in the >> environment? >> Jouni >> >> PS.somehow the init.d shell scripts were easier to debug ... >> >> -- >> Jouni Ryn? mailto://Jouni.Ryno@REDACTED/ >> http://space.fmi.fi/~ryno/ >> Finnish Meteorological Institute http://www.fmi.fi/ >> Observation services >> P.O.BOX 503 Tel (+358)-29-5394656 >> FI-00101 Helsinki FAX (+358)-29-5395703 >> Finland priv-GSM (+358)-50-5302903 >> >> md5sum(the Ultimate Question of Life, The Universe, and Everything) = >> 0000000000000000000000000000002A >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 931 9877991 -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Nov 1 11:47:32 2017 From: zxq9@REDACTED (zxq9) Date: Wed, 01 Nov 2017 19:47:32 +0900 Subject: [erlang-questions] Dialyzer and numeric range values Message-ID: <4930013.OF25b72dTy@changa> If I give a typespec -type foo() :: 10..14. Dialyzer will show me 10 | 11 | 12 | 13 | 14 I can see this is inclusive. If I give a typespec like 4..65 Dialyzer will show me 1..255 If I give a typespec 3..500 Dialyzer will show me 1..1114111 Which is hugely different than what I want. Which also winds up meaning that 0..16#ffff, which should be the legal range for ports, winds up being interpreted as type char(), which happens to be 0..16#10ffff which is totally bizarre. In fact, the docs for inet show me this definition: ip6_address() = {0..65535, 0..65535, 0..65535, 0..65535, 0..65535, 0..65535, 0..65535, 0..65535} ( http://erlang.org/doc/man/inet.html ) which I am shown Dialyzer interpreting as {char(), char(), char(), char(), char(), char(), char(), char()} and I can see the actual definition of char in the docs is indeed -type char() :: 0..16#10ffff. ( erlang.org/documentation/doc-5.8/doc/reference_manual/typespec.html ) What is going on here? Is this a new thing with R20's Dialyzer, or have I just never noticed these multi-byte leaps in value range resolution? -Craig From mononcqc@REDACTED Wed Nov 1 12:27:35 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 1 Nov 2017 07:27:35 -0400 Subject: [erlang-questions] Dialyzer and numeric range values In-Reply-To: <4930013.OF25b72dTy@changa> References: <4930013.OF25b72dTy@changa> Message-ID: <20171101112734.GA23845@ferdmbp.local> On 11/01, zxq9 wrote: >What is going on here? Is this a new thing with R20's Dialyzer, or have >I just never noticed these multi-byte leaps in value range resolution? > You never noticed it. Dialyzer kind of keeps the right to expand ranges arbitrarily. You can see in the source code that anything set of types above 12 gets merged into one: https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393c426b62f67/lib/hipe/cerl/erl_types.erl#L1997-L2011 The same kind of stuff also happens with atoms or even types in general (https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393c426b62f67/lib/hipe/cerl/erl_types.erl#L4860-L4866) which can be merged into the any() type. It's frankly a bit of a bummer but I figure it's something Dialyzer does to be less memory-hungry, at the cost of accuracy. From zxq9@REDACTED Wed Nov 1 16:03:46 2017 From: zxq9@REDACTED (zxq9) Date: Thu, 02 Nov 2017 00:03:46 +0900 Subject: [erlang-questions] Dialyzer and numeric range values In-Reply-To: <20171101112734.GA23845@ferdmbp.local> References: <4930013.OF25b72dTy@changa> <20171101112734.GA23845@ferdmbp.local> Message-ID: <1932524.ZpThSqd2xx@changa> On 2017?11?01? ??? 07:27:35 Fred Hebert wrote: > On 11/01, zxq9 wrote: > >What is going on here? Is this a new thing with R20's Dialyzer, or have > >I just never noticed these multi-byte leaps in value range resolution? > > You never noticed it. Dialyzer kind of keeps the right to expand ranges > arbitrarily. You can see in the source code that anything set of types > above 12 gets merged into one: > https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393c426b62f67/lib/hipe/cerl/erl_types.erl#L1997-L2011 > > The same kind of stuff also happens with atoms or even types in general > (https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393c426b62f67/lib/hipe/cerl/erl_types.erl#L4860-L4866) > which can be merged into the any() type. > > It's frankly a bit of a bummer but I figure it's something Dialyzer does > to be less memory-hungry, at the cost of accuracy. Hah! Wow. That's a pretty darn steep range curve on numerics! Incidentally, also one of the only apt uses of an `if` I've seen in a while. I've stayed away from Dialyzer internals for a reason (I get sort of obsessed about such things) but I wonder if it wouldn't be possible to establish numeric ranges as test ranges instead of enumerations? It seems like this is the case, actually, but obviously I am missing something if they can't set ?int_range(X, Y) directly and must instead only ever set either ?int_range(1, ?MAX_BYTE) or ?int_range(1, ?MAX_CHAR). https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393c426b62f67/lib/hipe/cerl/erl_types.erl#L342 The only place this approach seems to hold is called t_from_range_unsafe/2. https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393c426b62f67/lib/hipe/cerl/erl_types.erl#L2018 I assume "unsafe" for a reason, though the expected tests actually do exist. https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393c426b62f67/lib/hipe/cerl/erl_types.erl#L2101 Hm. So I'm just puzzled at the reasoning behind the need to enumerate ranges, and then the presence code that doesn't need ranges to be enumerated. Even more to the point, why not a list of arbitrary ranges? Too slow? Hard to imagine speed is the primary concern. Perhaps simply some work that never quite got polished off because, well, Dialyzer is already super useful in 90% of cases? Anyway, thanks for pointing that out. Quite interesting. -Craig From seriy.pr@REDACTED Wed Nov 1 16:15:19 2017 From: seriy.pr@REDACTED (=?UTF-8?B?0KHQtdGA0LPQtdC5INCf0YDQvtGF0L7RgNC+0LI=?=) Date: Wed, 1 Nov 2017 16:15:19 +0100 Subject: [erlang-questions] How does Erlang TCP determine the end of a TCP stream? Message-ID: You may look at epgsql's packet parser as an example. It has packet-type + packet-length - prefixed packet structure and it doesn't use {packet, N} option. handle_info({_, Sock, Data2}, ...) https://github.com/epgsql/epgsql/blob/3.2.0/src/epgsql_sock.erl#L152 tail buffering https://github.com/epgsql/epgsql/blob/3.2.0/src/epgsql_sock.erl#L359 here we check packet bounds https://github.com/epgsql/epgsql/blob/3.2.0/src/epgsql_wire.erl#L22 > Stanislaw, > Is there a common solution to this problem? Erlang was built to be used in > servers/switches, i would assume that there was a solution built into OTP? > If not, do you handle it just at the application later with gen_tcp:recv() > and read X bytes at a time from the socket based on delimiters and such? > How would you reconcile this with using a gen_server implementation that > receives messages from the ssl socket in handle info ? > I appreciate the help, I?m happy I didn?t roll this further -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Wed Nov 1 23:02:04 2017 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Wed, 1 Nov 2017 23:02:04 +0100 Subject: [erlang-questions] mnesia bound keys In-Reply-To: References: Message-ID: Consider a key-value store. You might have records -record(kv, {key, val}). which would be stored as 3-tuples {kv, Key, Val} in a mnesia table kv. A call mnesia:match_object({kv, 42, '_'}) has a pattern where the key position is bound to an actual value (42) but the value position is unbound ('_' is the wildcard which matches anything). This call would then find all records in the kv table having key 42. On the other hand, a call mnesia:match_object({kv, '_', foo}) has a pattern where the key position is unbound but the value position is bound to the value foo. This call would then find all records in the kv table having value foo. The significance of the key is that it's the _index_ for the table, meaning that any access to the table that supplies the key can quickly locate where the relevant records are stored. Accesses that do not supply the key generally have to enumerate all records and inspect them to see which ones match the pattern. This means that accesses with a key are much quicker than those without. On Tue, Oct 31, 2017 at 10:48 PM, Karlo Kuna wrote: > i still don't fully grasp the concept. short example would be > greatly appreciated. > > On Tue, Oct 31, 2017 at 8:15 PM, Mikael Pettersson > wrote: > >> Presumably you're referring to mnesia:match_object/1 or /3. It means >> that the key position of the pattern is a ground value and not a variable. >> >> On Tue, Oct 31, 2017 at 7:24 PM, Karlo Kuna wrote: >> >>> i was wondering what exactly "key is bound" means in context of mnesia >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tobias.Schlager@REDACTED Thu Nov 2 09:10:09 2017 From: Tobias.Schlager@REDACTED (Tobias Schlager) Date: Thu, 2 Nov 2017 08:10:09 +0000 Subject: [erlang-questions] run_erl and systemd In-Reply-To: <1509433072.2295.97.camel@fmi.fi> References: <1509433072.2295.97.camel@fmi.fi> Message-ID: <12F2115FD1CCEE4294943B2608A18FA301C2A0DD35@MAIL01.win.lbaum.eu> Hi Jouni, we dropped the relx/rebar start script altogether and are now using erlexec directly in our service files. See [1] for an example. It has the drawback that you must set all environment variables needed by the emulator yourself. Regards Tobias Schlager [1] https://github.com/lindenbaum/http2smtp/blob/master/config/http2smtp.service.template From Ingela.Anderton.Andin@REDACTED Thu Nov 2 12:47:39 2017 From: Ingela.Anderton.Andin@REDACTED (Ingela Anderton Andin) Date: Thu, 2 Nov 2017 12:47:39 +0100 Subject: [erlang-questions] Patch package OTP 20.1.4 released Message-ID: Patch Package:?????????? OTP 20.1.4 Git Tag:?????????????????????? OTP-20.1.4 Date:????????????????????????? 2017-11-02 Trouble Report Id:????? OTP-14656, OTP-14696, OTP-14716 Seq num: System:????????????????????? OTP Release:???????????????????? 20 Application:?????????????? inets-6.4.3 Predecessor:???????????? OTP 20.1.3 ?Check out the git tag OTP-20.1.4, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- inets-6.4.3 ----------------------------------------------------- ?--------------------------------------------------------------------- ?The inets-6.4.3 application can be applied independently of other ?applications on a full OTP 20 installation. ?--- Improvements and New Features --- ? OTP-14656??? Application(s): inets ?????????????? Fix broken handling of POST requests ?????????????? New chunk mechanism of body data in POST requests added ?????????????? in 5d01c70ca399edf28e99dc760506329689fab6ba broke ?????????????? handling of POST body data not using the new mechanism. ? OTP-14696??? Application(s): inets ?????????????? Make sure ints:stop/2 of the service httpd is ?????????????? synchronous ? OTP-14716??? Application(s): inets ?????????????? Honor status code returned by ESI script and modernize ?????????????? "location" header handling. ?Full runtime dependencies of inets-6.4.3: erts-6.0, kernel-3.0, ?mnesia-4.12, runtime_tools-1.8.14, ssl-5.3.4, stdlib-2.0 From frank.muller.erl@REDACTED Thu Nov 2 15:48:42 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 02 Nov 2017 14:48:42 +0000 Subject: [erlang-questions] Beam.smp killed by Linux Message-ID: Hi guys, We just found this in our systemd?s logs: Nov 02 15:17:01 ns342284.ip-91-121-153.eu CROND[24089]: (root) CMD (/usr/local/rtm/bin/rtm 55 > /dev/null 2> /dev/null) Nov 02 15:17:53 ns342284.ip-91-121-153.eu kernel: beam.smp[24236]: segfault at 1c ip 000000000051be70 sp 00007f2d3e0bfea0 error 6 in beam.smp[400000+22f000] Nov 02 15:17:53 ns342284.ip-91-121-153.eu systemd[1]: zeta.service: main process exited, code=killed, status=11/SEGV Nov 02 15:17:53 ns342284.ip-91-121-153.eu systemd[1]: zeta.service: control process exited, code=exited status=1 Anyone knows what may cause this? Machine: CentOS 7.2.1511 Kernel: 3.10.0.el7.x86_64 Erlang: 16B03-1 CPU: 4 Intel Xeon(R) E5-2690 @2.60GHz RAM: 16GB Disk: 2TB Best /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke@REDACTED Thu Nov 2 16:02:54 2017 From: luke@REDACTED (Luke Bakken) Date: Thu, 2 Nov 2017 08:02:54 -0700 Subject: [erlang-questions] Beam.smp killed by Linux In-Reply-To: References: Message-ID: Hi Frank, That message means that beam.smp died from a segmentation fault, and systemd is reporting it in the log. There are many things that can cause this and that have been fixed in later Erlang/OTP releases. You're using a very old release so my first recommendation would be to upgrade if possible. Tracking down the root cause of this requires building a VM with debug symbols and getting a core file from the next crash. Good luck - Luke On Thu, Nov 2, 2017 at 7:48 AM, Frank Muller wrote: > Hi guys, > > We just found this in our systemd?s logs: > > Nov 02 15:17:01 ns342284.ip-91-121-153.eu CROND[24089]: (root) CMD > (/usr/local/rtm/bin/rtm 55 > /dev/null 2> /dev/null) > Nov 02 15:17:53 ns342284.ip-91-121-153.eu kernel: beam.smp[24236]: segfault > at 1c ip 000000000051be70 sp 00007f2d3e0bfea0 error 6 in > beam.smp[400000+22f000] > Nov 02 15:17:53 ns342284.ip-91-121-153.eu systemd[1]: zeta.service: main > process exited, code=killed, status=11/SEGV > Nov 02 15:17:53 ns342284.ip-91-121-153.eu systemd[1]: zeta.service: control > process exited, code=exited status=1 > > Anyone knows what may cause this? > > Machine: CentOS 7.2.1511 Kernel: 3.10.0.el7.x86_64 > Erlang: 16B03-1 > CPU: 4 Intel Xeon(R) E5-2690 @2.60GHz > RAM: 16GB > Disk: 2TB > > Best > /Frank > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From g@REDACTED Fri Nov 3 01:54:06 2017 From: g@REDACTED (Guilherme Andrade) Date: Fri, 3 Nov 2017 00:54:06 +0000 Subject: [erlang-questions] Beam.smp killed by Linux In-Reply-To: References: Message-ID: On 2 November 2017 at 15:02, Luke Bakken wrote: > Tracking down the root cause of this requires building a VM with debug > symbols and getting a core file from the next crash. > In case that's unpractical, objdump'ing the beam.smp binary and navigating to the location of the crash might also help if the misbehaving code is listed under a routine with a readable name and that name can be traced back to the ERTS C code. -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Nov 3 09:42:10 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 03 Nov 2017 08:42:10 +0000 Subject: [erlang-questions] Beam.smp killed by Linux In-Reply-To: References: Message-ID: Thanks guys. I?ll try to recompile with 20.1 + debug symbols on and I?ll leg you know. /Frank On 2 November 2017 at 15:02, Luke Bakken wrote: > >> Tracking down the root cause of this requires building a VM with debug >> symbols and getting a core file from the next crash. >> > > In case that's unpractical, objdump'ing the beam.smp binary and navigating > to the location of the crash might also help if the misbehaving code is > listed under a routine with a readable name and that name can be traced > back to the ERTS C code. > > -- > Guilherme > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Fri Nov 3 11:06:16 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 03 Nov 2017 10:06:16 +0000 Subject: [erlang-questions] Beam.smp killed by Linux In-Reply-To: References: Message-ID: Typical errors include: * Bug in the VM * Hardware error, especially if running on older hardware * NIF has a bug * Linked in driver has a bug On Fri, Nov 3, 2017 at 9:42 AM Frank Muller wrote: > Thanks guys. > > I?ll try to recompile with 20.1 + debug symbols on and I?ll leg you know. > > /Frank > > On 2 November 2017 at 15:02, Luke Bakken wrote: >> >>> Tracking down the root cause of this requires building a VM with debug >>> symbols and getting a core file from the next crash. >>> >> >> In case that's unpractical, objdump'ing the beam.smp binary and >> navigating to the location of the crash might also help if the misbehaving >> code is listed under a routine with a readable name and that name can be >> traced back to the ERTS C code. >> >> -- >> Guilherme >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Nov 3 11:34:17 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 03 Nov 2017 10:34:17 +0000 Subject: [erlang-questions] Beam.smp killed by Linux In-Reply-To: References: Message-ID: Hi Jesper, We do not use any NIF nor linked in driver. But I agree R16B03-1 is quite old (porting our app now to 20.1). I?ll try to change to a newer hardware if 20.1 will not solve the issue. Thanks /Frank Typical errors include: > > * Bug in the VM > * Hardware error, especially if running on older hardware > * NIF has a bug > * Linked in driver has a bug > > > > On Fri, Nov 3, 2017 at 9:42 AM Frank Muller > wrote: > >> Thanks guys. >> >> I?ll try to recompile with 20.1 + debug symbols on and I?ll leg you know. >> >> /Frank >> >> On 2 November 2017 at 15:02, Luke Bakken wrote: >>> >>>> Tracking down the root cause of this requires building a VM with debug >>>> symbols and getting a core file from the next crash. >>>> >>> >>> In case that's unpractical, objdump'ing the beam.smp binary and >>> navigating to the location of the crash might also help if the misbehaving >>> code is listed under a routine with a readable name and that name can be >>> traced back to the ERTS C code. >>> >>> -- >>> Guilherme >>> >> _______________________________________________ > > >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Fri Nov 3 11:47:09 2017 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 3 Nov 2017 10:47:09 +0000 Subject: [erlang-questions] SSL 'verify_peer' client option changed between Erlang 19.3/20.1? Message-ID: I've got some test code where I connect an Erlang ssl client to an Erlang ssl server on localhost. On Erlang 19.3, it was passing fine. On Erlang 20.1, it started failing with {bad_cert,hostname_check_failed}. Investigation reveals that I'm connecting to "localhost", the server cert has ".../CN=testserver", and I'm passing {verify, verify_peer} in the client options. My question is, basically: why didn't Erlang 19 fail? From ingela.andin@REDACTED Fri Nov 3 15:43:38 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 3 Nov 2017 15:43:38 +0100 Subject: [erlang-questions] SSL 'verify_peer' client option changed between Erlang 19.3/20.1? In-Reply-To: References: Message-ID: Hi! In OTP 20, TLS client processes will by default call public_key:pkix_verify_hostname/2 to verify the hostname of the connection with the server certificates specified hostname during certificate path validation. The user may explicitly disables it. OTP 19 did not perform this check, it was left up to the application to perform it in the verify_fun if they wanted to. It is not really part of the TLS protocol but it is mandated that TLS client perform the check. Regards Ingela Erlang/OTP team - Ericsson AB 2017-11-03 11:47 GMT+01:00 Roger Lipscombe : > I've got some test code where I connect an Erlang ssl client to an > Erlang ssl server on localhost. On Erlang 19.3, it was passing fine. > On Erlang 20.1, it started failing with > {bad_cert,hostname_check_failed}. > > Investigation reveals that I'm connecting to "localhost", the server > cert has ".../CN=testserver", and I'm passing {verify, verify_peer} in > the client options. > > My question is, basically: why didn't Erlang 19 fail? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Nov 3 16:16:03 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 03 Nov 2017 15:16:03 +0000 Subject: [erlang-questions] SSL 'verify_peer' client option changed between Erlang 19.3/20.1? In-Reply-To: References: Message-ID: Ingela, Couldn?t find out how to disable this option. Can you point us to it please ? /Frank Hi! > > In OTP 20, TLS client processes will by default call > public_key:pkix_verify_hostname/2 to verify the hostname of the connection > with the server certificates specified hostname during certificate path > validation. The user may explicitly disables it. OTP 19 did not perform > this check, it was left up to the application to perform it in the > verify_fun if they wanted to. It is not really part of the TLS protocol but > it is mandated that TLS client perform the check. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > 2017-11-03 11:47 GMT+01:00 Roger Lipscombe : > >> I've got some test code where I connect an Erlang ssl client to an >> Erlang ssl server on localhost. On Erlang 19.3, it was passing fine. >> On Erlang 20.1, it started failing with >> {bad_cert,hostname_check_failed}. >> >> Investigation reveals that I'm connecting to "localhost", the server >> cert has ".../CN=testserver", and I'm passing {verify, verify_peer} in >> the client options. >> >> My question is, basically: why didn't Erlang 19 fail? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri Nov 3 16:53:42 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 3 Nov 2017 16:53:42 +0100 Subject: [erlang-questions] SSL 'verify_peer' client option changed between Erlang 19.3/20.1? In-Reply-To: References: Message-ID: {server_name_indication, hostname() | disable} 2017-11-03 16:16 GMT+01:00 Frank Muller : > Ingela, > > Couldn?t find out how to disable this option. > Can you point us to it please ? > > /Frank > > Hi! >> >> In OTP 20, TLS client processes will by default call >> public_key:pkix_verify_hostname/2 to verify the hostname of the >> connection with the server certificates specified hostname during >> certificate path validation. The user may explicitly disables it. OTP 19 >> did not perform this check, it was left up to the application to perform it >> in the verify_fun if they wanted to. It is not really part of the TLS >> protocol but it is mandated that TLS client perform the check. >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> 2017-11-03 11:47 GMT+01:00 Roger Lipscombe : >> >>> I've got some test code where I connect an Erlang ssl client to an >>> Erlang ssl server on localhost. On Erlang 19.3, it was passing fine. >>> On Erlang 20.1, it started failing with >>> {bad_cert,hostname_check_failed}. >>> >>> Investigation reveals that I'm connecting to "localhost", the server >>> cert has ".../CN=testserver", and I'm passing {verify, verify_peer} in >>> the client options. >>> >>> My question is, basically: why didn't Erlang 19 fail? >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Nov 3 19:18:52 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 3 Nov 2017 21:18:52 +0300 Subject: [erlang-questions] SSL 'verify_peer' client option changed between Erlang 19.3/20.1? In-Reply-To: References: Message-ID: heh, all tests with https://127.0.0.1:5672 got broken =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sunboshan@REDACTED Sun Nov 5 04:39:58 2017 From: sunboshan@REDACTED (Boshan Sun) Date: Sat, 4 Nov 2017 20:39:58 -0700 Subject: [erlang-questions] Erlang C NIF access to process struct Message-ID: Hi all, In Erlang, you can use `hipe_bifs:show_heap(self()).` to debug the process's heap. From: 0x000000001ee5c6e0 to 0x000000001ee5c748 | H E A P | | Address | Contents | |--------------------|--------------------| 4 | 0x000000001ee5c6e0 | 0x000000000000004f | [] | 0x000000001ee5c6e8 | 0xfffffffffffffffb | 3 | 0x000000001ee5c6f0 | 0x000000000000003f | [4] | 0x000000001ee5c6f8 | 0x000000001ee5c6e1 | |--------------------|--------------------| However the implementation is for 32-bits machine, and the table is kind of off for a 64-bits erlang. I want to write a C NIF to print similar heap information. What I need is the address of `process -> heap` and the heap size so I can get the heap's information. My question is, is there a way to get a pointer to the given process via pid in C NIF? In NIF doc there's enif_get_local_pid , how does this pid translate to the process pointer? Thanks! Boshan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Sun Nov 5 10:34:47 2017 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Sun, 5 Nov 2017 10:34:47 +0100 Subject: [erlang-questions] Erlang C NIF access to process struct In-Reply-To: References: Message-ID: The "off" table is due to a glitch in the output routine. The too short "H E A P" banner is indeed a 32-vs-64 bit bug. I fixed both in https://github.com/erlang/otp/pull/1619. WRT accessing the current "struct process", I haven't looked in detail, but in general the NIF interface wants to abstract away representational details and replace them function calls (actual runtime function calls, not just function-like macros). I don't see direct accesses to "struct process" fitting into that. On Sun, Nov 5, 2017 at 4:39 AM, Boshan Sun wrote: > Hi all, > > In Erlang, you can use `hipe_bifs:show_heap(self()).` to debug the > process's heap. > > From: 0x000000001ee5c6e0 to 0x000000001ee5c748 > | H E A P | > | Address | Contents | > |--------------------|--------------------| > 4 | 0x000000001ee5c6e0 | 0x000000000000004f | > [] | 0x000000001ee5c6e8 | 0xfffffffffffffffb | > 3 | 0x000000001ee5c6f0 | 0x000000000000003f | > [4] | 0x000000001ee5c6f8 | 0x000000001ee5c6e1 | > |--------------------|--------------------| > > However the implementation is for 32-bits machine, and the table is kind > of off for a 64-bits erlang. > > I want to write a C NIF to print similar heap information. What I need is > the address of `process -> heap` and the heap size so I can get the > heap's information. > > My question is, is there a way to get a pointer to the given process via > pid in C NIF? > > In NIF doc there's enif_get_local_pid > , how does > this pid translate to the process pointer? > > Thanks! > > Boshan > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sun Nov 5 17:44:34 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sun, 5 Nov 2017 17:44:34 +0100 Subject: [erlang-questions] map elements in defined order In-Reply-To: <098c5d52-5435-8c77-9fad-8c8247447034@cs.otago.ac.nz> References: <11a6bcc6-8fc7-0863-98de-ac071fdcc8b0@cs.otago.ac.nz> <018e93d8-a53d-c5e9-c920-57e7c7086e68@ericsson.com> <098c5d52-5435-8c77-9fad-8c8247447034@cs.otago.ac.nz> Message-ID: > On 30 Oct 2017, at 01:16, Richard A. O'Keefe wrote: > > For term_to_binary/2, of course the result > depends on the Options argument, but I take > it now that even being explicit about the > Options is not enough. Can we have a > 'canonical' option? > +1 . That would be an awesome option :) - benoit From sunboshan@REDACTED Mon Nov 6 04:55:12 2017 From: sunboshan@REDACTED (Boshan Sun) Date: Sun, 5 Nov 2017 19:55:12 -0800 Subject: [erlang-questions] Erlang C NIF access to process struct In-Reply-To: References: Message-ID: Does your fix also fix the variable alignment issue as well? The variable is in the beginning of the next row now, and should be in the end of same row. Also can you fix hipe_bifs:show_estack/1, hipe_bifs:show_nstack/1, hipe_bifs:show_pcb/1? They all suffer the "off" issue. BTW, I just read your paper *A Staged Tag Scheme for Erlang* and it explains Erlang's tag scheme pretty well. Thanks for that. The paper was written in 2000 for R6B/R7A, just wondering do you guys have some updated paper regarding for current state of Erlang/OTP's tag scheme? Thanks! 2017-11-05 1:34 GMT-08:00 Mikael Pettersson : > The "off" table is due to a glitch in the output routine. The too short > "H E A P" banner is indeed a 32-vs-64 bit bug. I fixed both in > https://github.com/erlang/otp/pull/1619. > > WRT accessing the current "struct process", I haven't looked in detail, > but in general the NIF interface wants to abstract away representational > details and replace them function calls (actual runtime function calls, not > just function-like macros). I don't see direct accesses to "struct > process" fitting into that. > > On Sun, Nov 5, 2017 at 4:39 AM, Boshan Sun wrote: > >> Hi all, >> >> In Erlang, you can use `hipe_bifs:show_heap(self()).` to debug the >> process's heap. >> >> From: 0x000000001ee5c6e0 to 0x000000001ee5c748 >> | H E A P | >> | Address | Contents | >> |--------------------|--------------------| >> 4 | 0x000000001ee5c6e0 | 0x000000000000004f | >> [] | 0x000000001ee5c6e8 | 0xfffffffffffffffb | >> 3 | 0x000000001ee5c6f0 | 0x000000000000003f | >> [4] | 0x000000001ee5c6f8 | 0x000000001ee5c6e1 | >> |--------------------|--------------------| >> >> However the implementation is for 32-bits machine, and the table is kind >> of off for a 64-bits erlang. >> >> I want to write a C NIF to print similar heap information. What I need is >> the address of `process -> heap` and the heap size so I can get the >> heap's information. >> >> My question is, is there a way to get a pointer to the given process via >> pid in C NIF? >> >> In NIF doc there's enif_get_local_pid >> , how does >> this pid translate to the process pointer? >> >> Thanks! >> >> Boshan >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickp@REDACTED Mon Nov 6 07:10:19 2017 From: rickp@REDACTED (Rick Payne) Date: Mon, 6 Nov 2017 19:10:19 +1300 Subject: [erlang-questions] Pre-allocating memory Message-ID: <46870768-08F2-4513-9D74-F814FF1DE7FF@rossfell.co.uk> I'm running the beam under OSv - a unikernel that tries to be ABI compatible with Linux, modulo a few things (such as no fork). Unfortunately, I'm seeing lots of memory fragmentation with my workload, probably due to issues in OSv's memory management. I had thought there was a way to get erlang to preallocate memory in advance using super-carriers, such that i could then rely on the beam to manage memory and hopefully not suffer the same fragmentation issues. However, I am failing to make this work. From my reading of the erts_alloc page, I have tried setting +MMscs to 8192 (as I want to allocate 8GB). This doesn't seem to happen (looking at the OSv stats - the memory isn't allocated). Have I misunderstood? It could also be that some facilities are not supported by OSv (but it does support mmap for instance) - so a pointer as to what to check would be most useful. Cheers, Rick From dmytro.lytovchenko@REDACTED Mon Nov 6 08:52:22 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Mon, 6 Nov 2017 08:52:22 +0100 Subject: [erlang-questions] Erlang C NIF access to process struct In-Reply-To: References: Message-ID: You might enjoy reading this one http://beam-wisdoms.clau.se/en/latest/indepth-memory-layout.html And also this one https://github.com/happi/theBeamBook 2017-11-06 4:55 GMT+01:00 Boshan Sun : > Does your fix also fix the variable alignment issue as well? The variable > is in the beginning of the next row now, and should be in the end of same > row. Also can you fix hipe_bifs:show_estack/1, hipe_bifs:show_nstack/1, > hipe_bifs:show_pcb/1? They all suffer the "off" issue. > > BTW, I just read your paper *A Staged Tag Scheme for Erlang* and it > explains Erlang's tag scheme pretty well. Thanks for that. The paper was > written in 2000 for R6B/R7A, just wondering do you guys have some updated > paper regarding for current state of Erlang/OTP's tag scheme? > > Thanks! > > 2017-11-05 1:34 GMT-08:00 Mikael Pettersson : > >> The "off" table is due to a glitch in the output routine. The too short >> "H E A P" banner is indeed a 32-vs-64 bit bug. I fixed both in >> https://github.com/erlang/otp/pull/1619. >> >> WRT accessing the current "struct process", I haven't looked in detail, >> but in general the NIF interface wants to abstract away representational >> details and replace them function calls (actual runtime function calls, not >> just function-like macros). I don't see direct accesses to "struct >> process" fitting into that. >> >> On Sun, Nov 5, 2017 at 4:39 AM, Boshan Sun wrote: >> >>> Hi all, >>> >>> In Erlang, you can use `hipe_bifs:show_heap(self()).` to debug the >>> process's heap. >>> >>> From: 0x000000001ee5c6e0 to 0x000000001ee5c748 >>> | H E A P | >>> | Address | Contents | >>> |--------------------|--------------------| >>> 4 | 0x000000001ee5c6e0 | 0x000000000000004f | >>> [] | 0x000000001ee5c6e8 | 0xfffffffffffffffb | >>> 3 | 0x000000001ee5c6f0 | 0x000000000000003f | >>> [4] | 0x000000001ee5c6f8 | 0x000000001ee5c6e1 | >>> |--------------------|--------------------| >>> >>> However the implementation is for 32-bits machine, and the table is kind >>> of off for a 64-bits erlang. >>> >>> I want to write a C NIF to print similar heap information. What I need >>> is the address of `process -> heap` and the heap size so I can get the >>> heap's information. >>> >>> My question is, is there a way to get a pointer to the given process via >>> pid in C NIF? >>> >>> In NIF doc there's enif_get_local_pid >>> , how does >>> this pid translate to the process pointer? >>> >>> Thanks! >>> >>> Boshan >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Mon Nov 6 13:34:47 2017 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Mon, 6 Nov 2017 13:34:47 +0100 Subject: [erlang-questions] Erlang C NIF access to process struct In-Reply-To: References: Message-ID: My PR should fix both issues with show_heap/1. I wasn't aware the other show_* BIFs had issues, I may take a look at them soonish. There is no updated document for the tag scheme AFAIK, but reading erts/emulator/beam/erl_term.h should, if you can decipher all #ifdefs and macros, provide the current details. The general principles remain the same, but new types and alternatives have appeared, and existing ones may have been reassigned. On Mon, Nov 6, 2017 at 4:55 AM, Boshan Sun wrote: > Does your fix also fix the variable alignment issue as well? The variable > is in the beginning of the next row now, and should be in the end of same > row. Also can you fix hipe_bifs:show_estack/1, hipe_bifs:show_nstack/1, > hipe_bifs:show_pcb/1? They all suffer the "off" issue. > > BTW, I just read your paper *A Staged Tag Scheme for Erlang* and it > explains Erlang's tag scheme pretty well. Thanks for that. The paper was > written in 2000 for R6B/R7A, just wondering do you guys have some updated > paper regarding for current state of Erlang/OTP's tag scheme? > > Thanks! > > 2017-11-05 1:34 GMT-08:00 Mikael Pettersson : > >> The "off" table is due to a glitch in the output routine. The too short >> "H E A P" banner is indeed a 32-vs-64 bit bug. I fixed both in >> https://github.com/erlang/otp/pull/1619. >> >> WRT accessing the current "struct process", I haven't looked in detail, >> but in general the NIF interface wants to abstract away representational >> details and replace them function calls (actual runtime function calls, not >> just function-like macros). I don't see direct accesses to "struct >> process" fitting into that. >> >> On Sun, Nov 5, 2017 at 4:39 AM, Boshan Sun wrote: >> >>> Hi all, >>> >>> In Erlang, you can use `hipe_bifs:show_heap(self()).` to debug the >>> process's heap. >>> >>> From: 0x000000001ee5c6e0 to 0x000000001ee5c748 >>> | H E A P | >>> | Address | Contents | >>> |--------------------|--------------------| >>> 4 | 0x000000001ee5c6e0 | 0x000000000000004f | >>> [] | 0x000000001ee5c6e8 | 0xfffffffffffffffb | >>> 3 | 0x000000001ee5c6f0 | 0x000000000000003f | >>> [4] | 0x000000001ee5c6f8 | 0x000000001ee5c6e1 | >>> |--------------------|--------------------| >>> >>> However the implementation is for 32-bits machine, and the table is kind >>> of off for a 64-bits erlang. >>> >>> I want to write a C NIF to print similar heap information. What I need >>> is the address of `process -> heap` and the heap size so I can get the >>> heap's information. >>> >>> My question is, is there a way to get a pointer to the given process via >>> pid in C NIF? >>> >>> In NIF doc there's enif_get_local_pid >>> , how does >>> this pid translate to the process pointer? >>> >>> Thanks! >>> >>> Boshan >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Mon Nov 6 16:13:36 2017 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Mon, 6 Nov 2017 16:13:36 +0100 Subject: [erlang-questions] Pre-allocating memory In-Reply-To: <46870768-08F2-4513-9D74-F814FF1DE7FF@rossfell.co.uk> References: <46870768-08F2-4513-9D74-F814FF1DE7FF@rossfell.co.uk> Message-ID: By default mmap() only sets up address space, but doesn't necessarily bind that address space to actual RAM. Normally that binding is populated "on demand" via page faults. (There are various flags to mmap() to modify this behaviour.) Perhaps check if /proc/$pid/maps contains an 8GB anonymous mapping? (where $pid is the pid of your beam process) On Mon, Nov 6, 2017 at 7:10 AM, Rick Payne wrote: > > I'm running the beam under OSv - a unikernel that tries to be ABI > compatible with Linux, modulo a few things (such as no fork). > Unfortunately, I'm seeing lots of memory fragmentation with my workload, > probably due to issues in OSv's memory management. > > I had thought there was a way to get erlang to preallocate memory in > advance using super-carriers, such that i could then rely on the beam to > manage memory and hopefully not suffer the same fragmentation issues. > However, I am failing to make this work. > > From my reading of the erts_alloc page, I have tried setting +MMscs to > 8192 (as I want to allocate 8GB). This doesn't seem to happen (looking at > the OSv stats - the memory isn't allocated). Have I misunderstood? > > It could also be that some facilities are not supported by OSv (but it > does support mmap for instance) - so a pointer as to what to check would be > most useful. > > Cheers, > Rick > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sunboshan@REDACTED Mon Nov 6 18:29:30 2017 From: sunboshan@REDACTED (Boshan Sun) Date: Mon, 6 Nov 2017 09:29:30 -0800 Subject: [erlang-questions] Erlang C NIF access to process struct In-Reply-To: References: Message-ID: Yes Dmytro I've also enjoy reading your BEAM Wisdom series and Erik Stenman's BEAM book. I think you guy's work are very good resource out there for explaining BEAM, along with of course, the source code. Mikael thanks for fixing those hipe_bifs:show_* functions, that would really delight 64-bits Erlang users :) 2017-11-06 4:34 GMT-08:00 Mikael Pettersson : > My PR should fix both issues with show_heap/1. I wasn't aware the other > show_* BIFs had issues, I may take a look at them soonish. > > There is no updated document for the tag scheme AFAIK, but reading > erts/emulator/beam/erl_term.h should, if you can decipher all #ifdefs and > macros, provide the current details. The general principles remain the > same, but new types and alternatives have appeared, and existing ones may > have been reassigned. > > On Mon, Nov 6, 2017 at 4:55 AM, Boshan Sun wrote: > >> Does your fix also fix the variable alignment issue as well? The variable >> is in the beginning of the next row now, and should be in the end of same >> row. Also can you fix hipe_bifs:show_estack/1, hipe_bifs:show_nstack/1, >> hipe_bifs:show_pcb/1? They all suffer the "off" issue. >> >> BTW, I just read your paper *A Staged Tag Scheme for Erlang* and it >> explains Erlang's tag scheme pretty well. Thanks for that. The paper was >> written in 2000 for R6B/R7A, just wondering do you guys have some updated >> paper regarding for current state of Erlang/OTP's tag scheme? >> >> Thanks! >> >> 2017-11-05 1:34 GMT-08:00 Mikael Pettersson : >> >>> The "off" table is due to a glitch in the output routine. The too short >>> "H E A P" banner is indeed a 32-vs-64 bit bug. I fixed both in >>> https://github.com/erlang/otp/pull/1619. >>> >>> WRT accessing the current "struct process", I haven't looked in detail, >>> but in general the NIF interface wants to abstract away representational >>> details and replace them function calls (actual runtime function calls, not >>> just function-like macros). I don't see direct accesses to "struct >>> process" fitting into that. >>> >>> On Sun, Nov 5, 2017 at 4:39 AM, Boshan Sun wrote: >>> >>>> Hi all, >>>> >>>> In Erlang, you can use `hipe_bifs:show_heap(self()).` to debug the >>>> process's heap. >>>> >>>> From: 0x000000001ee5c6e0 to 0x000000001ee5c748 >>>> | H E A P | >>>> | Address | Contents | >>>> |--------------------|--------------------| >>>> 4 | 0x000000001ee5c6e0 | 0x000000000000004f | >>>> [] | 0x000000001ee5c6e8 | 0xfffffffffffffffb | >>>> 3 | 0x000000001ee5c6f0 | 0x000000000000003f | >>>> [4] | 0x000000001ee5c6f8 | 0x000000001ee5c6e1 | >>>> |--------------------|--------------------| >>>> >>>> However the implementation is for 32-bits machine, and the table is >>>> kind of off for a 64-bits erlang. >>>> >>>> I want to write a C NIF to print similar heap information. What I need >>>> is the address of `process -> heap` and the heap size so I can get the >>>> heap's information. >>>> >>>> My question is, is there a way to get a pointer to the given process >>>> via pid in C NIF? >>>> >>>> In NIF doc there's enif_get_local_pid >>>> , how does >>>> this pid translate to the process pointer? >>>> >>>> Thanks! >>>> >>>> Boshan >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Nov 6 23:05:55 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 7 Nov 2017 11:05:55 +1300 Subject: [erlang-questions] Job opportunities? In-Reply-To: References: Message-ID: I have just heard through my lawyer that the University has definitely decided to fire me, so now I am looking for work. Anyone know of anything that could be done by someone in Dunedin, New Zealand? From rickp@REDACTED Mon Nov 6 20:20:34 2017 From: rickp@REDACTED (Rick Payne) Date: Tue, 7 Nov 2017 08:20:34 +1300 Subject: [erlang-questions] Pre-allocating memory In-Reply-To: References: <46870768-08F2-4513-9D74-F814FF1DE7FF@rossfell.co.uk> Message-ID: <106343E6-9A72-4A3B-B008-314B60CF3104@rossfell.co.uk> > On 7 Nov 2017, at 04:13, Mikael Pettersson wrote: > > By default mmap() only sets up address space, but doesn't necessarily bind that address space to actual RAM. Normally that binding is populated "on demand" via page faults. (There are various flags to mmap() to modify this behaviour.) Ah, that could be it - it looks like adding MAP_POPULATE may help me. I'll see if modifying erts to do that will help... Cheers, Rick From denc716@REDACTED Tue Nov 7 18:54:18 2017 From: denc716@REDACTED (books) Date: Tue, 7 Nov 2017 12:54:18 -0500 Subject: [erlang-questions] Job opportunities? In-Reply-To: References: Message-ID: if you're asking for erlang related, may search on erlangcentral, and broaden your search with elixir as well, however most are in Europe or USA, if you'd like to move https://erlangcentral.org/jobs http://jobs.elixirdose.com/ Good luck. On Mon, Nov 6, 2017 at 5:05 PM, Richard A. O'Keefe wrote: > I have just heard through my lawyer that the University has definitely > decided to fire me, so now I am looking for work. Anyone know of > anything that could be done by someone in Dunedin, New Zealand? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Tue Nov 7 21:07:45 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 7 Nov 2017 21:07:45 +0100 Subject: [erlang-questions] canonical encoding for term_to_binary/binary_to_term ? Message-ID: <724A42DD-0042-4D62-807C-6D1C15ED974A@gmail.com> Following the recent discussion on maps ordering, I?m curious if someone has already worked on an implementation of a canonical term_to_binary/binary_to_term to be able to sign them and make sure we get the same data across the wire. Is there any lib that does that around? For now i?m just using a canonical version of JSON but it?s pretty inefficient ? - benoit From askjuise@REDACTED Tue Nov 7 23:25:35 2017 From: askjuise@REDACTED (Alexander Petrovsky) Date: Wed, 8 Nov 2017 01:25:35 +0300 Subject: [erlang-questions] rebar3 tar and endless loop Message-ID: Hi! I try to prepare erlang release, with follow config: {relx, [ > {release, {x, "0.0.1"}, > [x, > sasl]}, > {overlay, [ > {copy, "priv/*", "priv/"} > ... > ./rebar3 release But, relx doesn't support wildcard. I'm try the follow config: ... > {copy, "priv/", "priv/"} > ... > ./rebar3 release Unfortunately, the result will be *priv/priv*. But when I do: ... > {copy, "priv/", "./"} > ... > ./rebar3 release Everything goes without problems. But, when I'm try to make tarball from that here problems come: the *./rebar3 tar* goes to endless loop continuously add *x-0.0.1.tar.gz* to himself. The *./rebar3 tar* keep archive tar.gz file in the same directory with generated erlang release files. So, when rebar3 sees mappings like {copy, "priv/", "./"} it's transforms them into something like [{"./", "/Users/juise/Documents/x/_build/default/rel/x"}, > {"log/sasl", "/Users/juise/Documents/x/_build/default/rel/x/log/sasl"}] which cause endless loop. I see multiple solutions: - ./rebar3 tar must archive all files in release directory, and not pay attention to overlay directive; - relx should ignore mappings like {"./", "/Users/juise/Documents/x/_build/default/rel/x"}; - erl_tar should explicitly ignore result archive and don't add to himself. -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 931 9877991 -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Tue Nov 7 23:51:27 2017 From: g@REDACTED (Guilherme Andrade) Date: Tue, 7 Nov 2017 22:51:27 +0000 Subject: [erlang-questions] canonical encoding for term_to_binary/binary_to_term ? In-Reply-To: <724A42DD-0042-4D62-807C-6D1C15ED974A@gmail.com> References: <724A42DD-0042-4D62-807C-6D1C15ED974A@gmail.com> Message-ID: The ETF format[1] is fairly straight forward so, performance considerations excluded, is shouldn't be too hard to code a custom encoder which creates a bijective mapping between terms and binaries. A while back I worked a bit on an ETF decoder with customizable restrictions on which types to accept to decode (and which to reject); I'm sharing it[2] in case someone finds it enlightening. I found the trickier cases to be pids, ports, references and anonymous functions. [1]: http://erlang.org/doc/apps/erts/erl_ext_dist.html [2]: https://github.com/g-andrade/restricted_etf/blob/master/src/restricted_etf.erl On 7 November 2017 at 20:07, Benoit Chesneau wrote: > Following the recent discussion on maps ordering, I?m curious if someone > has already worked on an implementation of a canonical > term_to_binary/binary_to_term to be able to sign them and make sure we get > the same data across the wire. > > Is there any lib that does that around? For now i?m just using a > canonical version of JSON but it?s pretty inefficient ? > > - benoit > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Nov 8 00:08:05 2017 From: zxq9@REDACTED (zxq9) Date: Wed, 08 Nov 2017 08:08:05 +0900 Subject: [erlang-questions] canonical encoding for term_to_binary/binary_to_term ? In-Reply-To: <724A42DD-0042-4D62-807C-6D1C15ED974A@gmail.com> References: <724A42DD-0042-4D62-807C-6D1C15ED974A@gmail.com> Message-ID: <50365468.uZl5quHWrl@changa> On 2017?11?07? ??? 21:07:45 Benoit Chesneau wrote: > Following the recent discussion on maps ordering, I?m curious if someone has already worked on an implementation of a canonical term_to_binary/binary_to_term to be able to sign them and make sure we get the same data across the wire. > > Is there any lib that does that around? For now i?m just using a canonical version of JSON but it?s pretty inefficient ? So far I've had success with converting everything to a canonical basic form of tuples and sorted lists. The trick there is that every message in a protocol (or document system, whatever) requires a known schema so that you can identify whether a list of things is a list that needs to be sorted, or is a text string that must never be sorted. Tagging elements with atoms works just fine for that going into the format and back out again, but my point is there is always a requirement for prior knowledge of the data. This hasn't really been much extra annoyance in practice because the system where I do this already has well defined schemas, so writing an importer/exporter isn't much work. It is a bit less trouble than defining an ASN.1 schema to serialize the data as DER, which is what we did before, and that'S guaranteed to work everywhere. This may be "inefficient" in some sense but its never been a bottleneck. Even with a BIF, the burden is always going to be on the programmer to canonicalize and mark strings prior to sending anything to a pre-sign canonical serialization function because there is no way a serializer can know the difference between the meanings of certain nested structures. -Craig From dmkolesnikov@REDACTED Wed Nov 8 08:13:42 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 8 Nov 2017 09:13:42 +0200 Subject: [erlang-questions] Cause heartbeat timeout Message-ID: <0F18499E-EFE0-4B3E-A729-A0D61419A9DC@gmail.com> Hello, I am trying to debug an issue of node termination by heartbeat timeout and its recovery. I am looking for advice about the reproduction of heartbeat timeout in controlled environment. What is the best approach to freeze Erlang node? Best Regards, Dmitry From dmkolesnikov@REDACTED Wed Nov 8 09:44:45 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 8 Nov 2017 10:44:45 +0200 Subject: [erlang-questions] rebar3 tar and endless loop In-Reply-To: References: Message-ID: <3605AA5B-7D8B-4D07-9ACF-C47CA37625DB@gmail.com> Hello, The private folder of applications are automatically added to the release. There are not need to add an overlay. You can move your root /priv folder under you application x. Best Regards, Dmitry > On 8 Nov 2017, at 0.25, Alexander Petrovsky wrote: > > Hi! > > I try to prepare erlang release, with follow config: > > {relx, [ > {release, {x, "0.0.1"}, > [x, > sasl]}, > {overlay, [ > {copy, "priv/*", "priv/"} > ... > ./rebar3 release > > But, relx doesn't support wildcard. I'm try the follow config: > > ... > {copy, "priv/", "priv/"} > ... > ./rebar3 release > > Unfortunately, the result will be priv/priv. But when I do: > > ... > {copy, "priv/", "./"} > ... > ./rebar3 release > > Everything goes without problems. But, when I'm try to make tarball from that here problems come: the ./rebar3 tar goes to endless loop continuously add x-0.0.1.tar.gz to himself. > > The ./rebar3 tar keep archive tar.gz file in the same directory with generated erlang release files. So, when rebar3 sees mappings like {copy, "priv/", "./"} it's transforms them into something like > > [{"./", "/Users/juise/Documents/x/_build/default/rel/x"}, > {"log/sasl", "/Users/juise/Documents/x/_build/default/rel/x/log/sasl"}] > > which cause endless loop. > > I see multiple solutions: > - ./rebar3 tar must archive all files in release directory, and not pay attention to overlay directive; > - relx should ignore mappings like {"./", "/Users/juise/Documents/x/_build/default/rel/x"}; > - erl_tar should explicitly ignore result archive and don't add to himself. > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 931 9877991 > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From skypexu@REDACTED Wed Nov 8 10:32:43 2017 From: skypexu@REDACTED (Skype Xu) Date: Wed, 8 Nov 2017 17:32:43 +0800 Subject: [erlang-questions] weak resource reference possible in nif ? Message-ID: <0348cae8-b460-ef7b-48a9-f35de4c863fd@gmail.com> Hi, Suppose a nif module is multi-threaded, we keep a pointer to a resource in the module's internal list, when the resource is no longer used by erlang process, erlang vm calls the resource's destructor, and we remove the pointer from internal list in the destructor, that's fine. However a problem arises when the internal list is also used by another thread created in the nif module, when some external events were triggered, the thread tries to use the resource pointer and sends it to a process in the vm, problem is there's a race condition between destructor and the internal thread, in the destructor we try to remove it from the internal list, at the time, the resource's reference count is already zero, and before the destructor is called, the internal thread sent the resource to a proecess, by calling enif_make_resource() to increase the count to 1, but anyway after the destructor returned, the vm will recycle the memory of the resource despites the reference count is 1 again, this leads to corrupted memory. We can not keep a resource by adding extra reference count in nif module, if we do, the resource's destructor will never be called even if not any erlang process is using the resource, because the reference count will never to dropped to 0. It would be perfect if we can keep a weak reference in nif like the one in C++ does, one can use weak_ptr to get back a shared_ptr object if the object is still using, otherwise a null pointer is returned.? if this is implemented, we can queue a notification to the internal thread and the thread will remove any data related to the resource.? enif_monitor_process() is not a solution because we want vm to recycle the resource when it is not used, not after a process crashed or exited. From dm.klionsky@REDACTED Wed Nov 8 10:49:26 2017 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 8 Nov 2017 12:49:26 +0300 Subject: [erlang-questions] Cause heartbeat timeout In-Reply-To: <0F18499E-EFE0-4B3E-A729-A0D61419A9DC@gmail.com> References: <0F18499E-EFE0-4B3E-A729-A0D61419A9DC@gmail.com> Message-ID: <5A02D326.3030604@gmail.com> I faced a situation like this two times. It was on a 2 CPUs EC2 instances. Heartbeat timeout was 45 secs. The first offender was zlib:gzip/1 and ~1GB file, the other was ++ over two long lists. Both operations took about 1 min to complete. My explanation is the heart's erlang part can't send the heartbeat to the heart port in time because either the scheduler it's bound to is busy or port I/O is busy/congested. Hope this helps. On 11/08/2017 10:13 AM, Dmitry Kolesnikov wrote: > Hello, > > I am trying to debug an issue of node termination by heartbeat timeout and its recovery. I am looking for advice about the reproduction of heartbeat timeout in controlled environment. What is the best approach to freeze Erlang node? > > Best Regards, > Dmitry > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- BR, Dmitry From dm.klionsky@REDACTED Wed Nov 8 10:53:47 2017 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Wed, 8 Nov 2017 12:53:47 +0300 Subject: [erlang-questions] Cause heartbeat timeout In-Reply-To: <5A02D326.3030604@gmail.com> References: <0F18499E-EFE0-4B3E-A729-A0D61419A9DC@gmail.com> <5A02D326.3030604@gmail.com> Message-ID: <5A02D42B.6030409@gmail.com> Correction. It was -- the second time. On 11/08/2017 12:49 PM, Dmitry Klionsky wrote: > I faced a situation like this two times. > It was on a 2 CPUs EC2 instances. Heartbeat timeout was 45 secs. > > The first offender was zlib:gzip/1 and ~1GB file, the other was ++ > over two long lists. Both operations took about 1 min to complete. > > My explanation is the heart's erlang part can't send the heartbeat to > the heart port in time because > either the scheduler it's bound to is busy or port I/O is busy/congested. > > Hope this helps. > > On 11/08/2017 10:13 AM, Dmitry Kolesnikov wrote: >> Hello, >> >> I am trying to debug an issue of node termination by heartbeat >> timeout and its recovery. I am looking for advice about the >> reproduction of heartbeat timeout in controlled environment. What is >> the best approach to freeze Erlang node? >> >> Best Regards, >> Dmitry >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -- BR, Dmitry From tobias.lindahl@REDACTED Wed Nov 8 11:05:32 2017 From: tobias.lindahl@REDACTED (Tobias Lindahl) Date: Wed, 8 Nov 2017 11:05:32 +0100 Subject: [erlang-questions] Dialyzer and numeric range values In-Reply-To: <1932524.ZpThSqd2xx@changa> References: <4930013.OF25b72dTy@changa> <20171101112734.GA23845@ferdmbp.local> <1932524.ZpThSqd2xx@changa> Message-ID: Types need to collapse to a higher level of abstraction at some point or the fixpoint iteration of the analysis might not terminate. Note that there are an infinite number of ranges. In order for the analysis to keep track of when to collapse the ranges instead of expanding them, the structure of the analysis would have to take into account some kind of history of range expansion (e.g., expand the range X number of times, then collapse). I spent quite some time on trying to use the ranges in Dialyzer in a better way, but as I recall it, the non-termination was the reason for collapsing early. We used a more precise range analysis in the HiPE compiler for optimization, and it is certainly possible to get good results for it, but it is not as straightforward as it might seem. 2017-11-01 16:03 GMT+01:00 zxq9 : > On 2017?11?01? ??? 07:27:35 Fred Hebert wrote: > > On 11/01, zxq9 wrote: > > >What is going on here? Is this a new thing with R20's Dialyzer, or have > > >I just never noticed these multi-byte leaps in value range resolution? > > > > You never noticed it. Dialyzer kind of keeps the right to expand ranges > > arbitrarily. You can see in the source code that anything set of types > > above 12 gets merged into one: > > https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393 > c426b62f67/lib/hipe/cerl/erl_types.erl#L1997-L2011 > > > > The same kind of stuff also happens with atoms or even types in general > > (https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393 > c426b62f67/lib/hipe/cerl/erl_types.erl#L4860-L4866) > > which can be merged into the any() type. > > > > It's frankly a bit of a bummer but I figure it's something Dialyzer does > > to be less memory-hungry, at the cost of accuracy. > > Hah! Wow. That's a pretty darn steep range curve on numerics! > > Incidentally, also one of the only apt uses of an `if` I've seen in a > while. > > I've stayed away from Dialyzer internals for a reason (I get sort of > obsessed about such things) but I wonder if it wouldn't be possible to > establish numeric ranges as test ranges instead of enumerations? > > It seems like this is the case, actually, but obviously I am missing > something if they can't set ?int_range(X, Y) directly and must instead only > ever set either ?int_range(1, ?MAX_BYTE) or ?int_range(1, ?MAX_CHAR). > https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393 > c426b62f67/lib/hipe/cerl/erl_types.erl#L342 > > The only place this approach seems to hold is called t_from_range_unsafe/2. > https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393 > c426b62f67/lib/hipe/cerl/erl_types.erl#L2018 > I assume "unsafe" for a reason, though the expected tests actually do > exist. > https://github.com/erlang/otp/blob/fe1df7fc6bf050cb6c9bbd99eb9393 > c426b62f67/lib/hipe/cerl/erl_types.erl#L2101 > > Hm. So I'm just puzzled at the reasoning behind the need to enumerate > ranges, and then the presence code that doesn't need ranges to be > enumerated. Even more to the point, why not a list of arbitrary ranges? Too > slow? Hard to imagine speed is the primary concern. Perhaps simply some > work that never quite got polished off because, well, Dialyzer is already > super useful in 90% of cases? > > Anyway, thanks for pointing that out. Quite interesting. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Nov 8 11:33:11 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 8 Nov 2017 13:33:11 +0300 Subject: [erlang-questions] weak resource reference possible in nif ? In-Reply-To: <0348cae8-b460-ef7b-48a9-f35de4c863fd@gmail.com> References: <0348cae8-b460-ef7b-48a9-f35de4c863fd@gmail.com> Message-ID: The main problem here is with reference count. You should use this mechanism and properly protect access to data with brackets inc/dec -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Wed Nov 8 11:36:42 2017 From: askjuise@REDACTED (Alexander Petrovsky) Date: Wed, 08 Nov 2017 10:36:42 +0000 Subject: [erlang-questions] rebar3 tar and endless loop In-Reply-To: <3605AA5B-7D8B-4D07-9ACF-C47CA37625DB@gmail.com> References: <3605AA5B-7D8B-4D07-9ACF-C47CA37625DB@gmail.com> Message-ID: Hi! Sorry, the priv folder was just for example, for other user folders the problem is still take a place. Would be great to hear rebar3 maintainers opinion. ??, 8 ????. 2017 ?. ? 11:44, Dmitry Kolesnikov : > Hello, > > The private folder of applications are automatically added to the release. > There are not need to add an overlay. You can move your root /priv folder > under you application x. > > Best Regards, > Dmitry > > > On 8 Nov 2017, at 0.25, Alexander Petrovsky wrote: > > Hi! > > I try to prepare erlang release, with follow config: > > {relx, [ >> {release, {x, "0.0.1"}, >> [x, >> sasl]}, >> {overlay, [ >> {copy, "priv/*", "priv/"} >> ... >> ./rebar3 release > > > But, relx doesn't support wildcard. I'm try the follow config: > > ... >> {copy, "priv/", "priv/"} >> ... >> ./rebar3 release > > > Unfortunately, the result will be *priv/priv*. But when I do: > > ... >> {copy, "priv/", "./"} >> ... >> ./rebar3 release > > > Everything goes without problems. But, when I'm try to make tarball from > that here problems come: the *./rebar3 tar* goes to endless loop > continuously add *x-0.0.1.tar.gz* to himself. > > The *./rebar3 tar* keep archive tar.gz file in the same directory with > generated erlang release files. So, when rebar3 sees mappings like {copy, > "priv/", "./"} it's transforms them into something like > > [{"./", "/Users/juise/Documents/x/_build/default/rel/x"}, >> {"log/sasl", "/Users/juise/Documents/x/_build/default/rel/x/log/sasl"}] > > > which cause endless loop. > > I see multiple solutions: > - ./rebar3 tar must archive all files in release directory, and not pay > attention to overlay directive; > - relx should ignore mappings like {"./", > "/Users/juise/Documents/x/_build/default/rel/x"}; > - erl_tar should explicitly ignore result archive and don't add to himself. > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 931 9877991 > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 931 9877991 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Wed Nov 8 11:37:04 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 8 Nov 2017 12:37:04 +0200 Subject: [erlang-questions] Cause heartbeat timeout In-Reply-To: <5A02D326.3030604@gmail.com> References: <0F18499E-EFE0-4B3E-A729-A0D61419A9DC@gmail.com> <5A02D326.3030604@gmail.com> Message-ID: <97A413A3-67EF-4AC7-86E7-BAF80E489398@gmail.com> Thank you for tips. I?ve not managed to reproduce it with lists. The node get killed due to OOM. - Dmitry. > On 8 Nov 2017, at 11.49, Dmitry Klionsky wrote: > > I faced a situation like this two times. > It was on a 2 CPUs EC2 instances. Heartbeat timeout was 45 secs. > > The first offender was zlib:gzip/1 and ~1GB file, the other was ++ over two long lists. Both operations took about 1 min to complete. > > My explanation is the heart's erlang part can't send the heartbeat to the heart port in time because > either the scheduler it's bound to is busy or port I/O is busy/congested. > > Hope this helps. > > On 11/08/2017 10:13 AM, Dmitry Kolesnikov wrote: >> Hello, >> >> I am trying to debug an issue of node termination by heartbeat timeout and its recovery. I am looking for advice about the reproduction of heartbeat timeout in controlled environment. What is the best approach to freeze Erlang node? >> >> Best Regards, >> Dmitry >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- > BR, > Dmitry > From john.hogberg@REDACTED Wed Nov 8 11:59:23 2017 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Wed, 8 Nov 2017 10:59:23 +0000 Subject: [erlang-questions] weak resource reference possible in nif ? In-Reply-To: <0348cae8-b460-ef7b-48a9-f35de4c863fd@gmail.com> References: <0348cae8-b460-ef7b-48a9-f35de4c863fd@gmail.com> Message-ID: <1510138763.25410.24.camel@ericsson.com> Hi, Why not let the resource hold nothing besides a shared_ptr to the actual data, and use weak_ptrs in the internal list? You'd need to allocate a new resource when sending notifications from that internal thread (Since the original resource could have been destroyed), but other than that it should be pretty straightforward unless I've misunderstood something. Regards, John H?gberg On ons, 2017-11-08 at 17:32 +0800, Skype Xu wrote: > Hi, > > Suppose a nif module is multi-threaded, we keep a pointer to a > resource? > in the > module's internal list, when the resource is no longer used by > erlang? > process, > erlang vm calls the resource's destructor, and we remove the pointer > from > internal list in the destructor, that's fine. > However a problem arises when the internal list is also used by > another? > thread > created in the nif module, when some external events were triggered, > the? > thread > tries to use the resource pointer and sends it to a process in the > vm,? > problem > is there's a race condition between destructor and the internal > thread, > in the destructor we try to remove it from the internal list, at the? > time, the > resource's reference count is already zero, and before the destructor > is? > called, > the internal thread sent the resource to a proecess, by calling > enif_make_resource() to increase the count to 1, but anyway after the > destructor returned, the vm will recycle the memory of the resource? > despites > the reference count is 1 again, this leads to corrupted memory. > > We can not keep a resource by adding extra reference count in nif > module, > if we do, the resource's destructor will never be called even if not > any? > erlang > process is using the resource, because the reference count will never > to > dropped to 0. > > It would be perfect if we can keep a weak reference in nif like the > one in > C++ does, one can use weak_ptr to get back a shared_ptr object if > the? > object > is still using, otherwise a null pointer is returned.? if this is? > implemented, we > can queue a notification to the internal thread and the thread will > remove > any data related to the resource.? enif_monitor_process() is not a > solution > because we want vm to recycle the resource when it is not used, not > after > a process crashed or exited. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From facundo.olano@REDACTED Wed Nov 8 15:44:45 2017 From: facundo.olano@REDACTED (Facundo Olano) Date: Wed, 8 Nov 2017 11:44:45 -0300 Subject: [erlang-questions] Refreshing ssl certificates in a cowboy application Message-ID: Hi! We are serving a cowboy web application with certbot/let's encrypt certificates which expire every 90 days. Since the cert files are changing so often, I'm wondering if there's a way to reload them without having to restart the application. I understand cowboy ultimately uses the Erlang ssl module, so I guess this is not a cowboy specific issue. Thanks, Facundo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Wed Nov 8 16:52:38 2017 From: askjuise@REDACTED (Alexander Petrovsky) Date: Wed, 8 Nov 2017 18:52:38 +0300 Subject: [erlang-questions] rebar3 tar and endless loop In-Reply-To: References: <3605AA5B-7D8B-4D07-9ACF-C47CA37625DB@gmail.com> Message-ID: 2017-11-08 16:25 GMT+03:00 Kostis Sagonas : > On 11/08/2017 11:36 AM, Alexander Petrovsky wrote: > >> Would be great to hear rebar3 maintainers opinion. >> > > Why don't you send this as an issue in rebar3's repo then? > > Kostis > It's not only rebar3 problem. If you read the letter, you should notice, the problem occur in erl_tar. -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 931 9877991 -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Wed Nov 8 16:54:51 2017 From: z@REDACTED (Danil Zagoskin) Date: Wed, 8 Nov 2017 18:54:51 +0300 Subject: [erlang-questions] Refreshing ssl certificates in a cowboy application In-Reply-To: References: Message-ID: Hi! OTP's ssl subsystem with default settings reloads the certfile without any extra actions. See ssl_pem_cache.erl for implementation details. On Wed, Nov 8, 2017 at 5:44 PM, Facundo Olano wrote: > Hi! > > We are serving a cowboy web application with certbot/let's encrypt > certificates which expire every 90 days. Since the cert files are changing > so often, I'm wondering if there's a way to reload them without having to > restart the application. > > I understand cowboy ultimately uses the Erlang ssl module, so I guess this > is not a cowboy specific issue. > > Thanks, > Facundo. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Wed Nov 8 17:12:20 2017 From: t@REDACTED (Tristan Sloughter) Date: Wed, 08 Nov 2017 08:12:20 -0800 Subject: [erlang-questions] rebar3 tar and endless loop In-Reply-To: References: <3605AA5B-7D8B-4D07-9ACF-C47CA37625DB@gmail.com> Message-ID: <1510157540.199886.1165880144.4BC2A083@webmail.messagingengine.com> So the issue is simply the use of copying to `./` as opposed to `./somedir`, right? Certainly an issue, easy fix is to put it under another diectory, since you can't access it with `code:priv_dir` anyway since it isn't under an application no reason not to do `{copy, "priv", "etc/"}` but you probably already did that. I'd like to see the fix, if possible, in erl_tar but I'd have to take a closer look to know for sure if that is possible and what it would be. Not sure when I'll get time to dig in. -- Tristan Sloughter "I am not a crackpot" - Abe Simpson t@REDACTED On Wed, Nov 8, 2017, at 07:52 AM, Alexander Petrovsky wrote: > > > 2017-11-08 16:25 GMT+03:00 Kostis Sagonas : >> On 11/08/2017 11:36 AM, Alexander Petrovsky wrote: >>> Would be great to hear rebar3 maintainers opinion.>> >> Why don't you send this as an issue in rebar3's repo then? >> >> Kostis> > It's not only rebar3 problem. If you read the letter, you should > notice, the problem occur in erl_tar.> > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 931 9877991 > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Wed Nov 8 17:46:22 2017 From: askjuise@REDACTED (Alexander Petrovsky) Date: Wed, 8 Nov 2017 19:46:22 +0300 Subject: [erlang-questions] rebar3 tar and endless loop In-Reply-To: <1510157540.199886.1165880144.4BC2A083@webmail.messagingengine.com> References: <3605AA5B-7D8B-4D07-9ACF-C47CA37625DB@gmail.com> <1510157540.199886.1165880144.4BC2A083@webmail.messagingengine.com> Message-ID: 2017-11-08 19:12 GMT+03:00 Tristan Sloughter : > So the issue is simply the use of copying to `./` as opposed to > `./somedir`, right? > Exactly! > > Certainly an issue, easy fix is to put it under another diectory, since > you can't access it with `code:priv_dir` anyway since it isn't under an > application no reason not to do `{copy, "priv", "etc/"}` but you probably > already did that. > > I'd like to see the fix, if possible, in erl_tar but I'd have to take a > closer look to know for sure if that is possible and what it would be. Not > sure when I'll get time to dig in. > I'm agree that it's will be perfect to fix the problem in erl_tar, but as quick fix we can ignore ./ mapping in https://github.com/erlware/relx/blob/master/src/rlx_prv_archive.erl#L111, and teach overlay_files ignore ./ mappings > > -- > Tristan Sloughter > "I am not a crackpot" - Abe Simpson > t@REDACTED > > > On Wed, Nov 8, 2017, at 07:52 AM, Alexander Petrovsky wrote: > > > > 2017-11-08 16:25 GMT+03:00 Kostis Sagonas : > > On 11/08/2017 11:36 AM, Alexander Petrovsky wrote: > > Would be great to hear rebar3 maintainers opinion. > > > Why don't you send this as an issue in rebar3's repo then? > > Kostis > > > It's not only rebar3 problem. If you read the letter, you should notice, > the problem occur in erl_tar. > > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 931 9877991 > > *_______________________________________________* > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 931 9877991 -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Wed Nov 8 17:54:32 2017 From: t@REDACTED (Tristan Sloughter) Date: Wed, 08 Nov 2017 08:54:32 -0800 Subject: [erlang-questions] rebar3 tar and endless loop In-Reply-To: References: <3605AA5B-7D8B-4D07-9ACF-C47CA37625DB@gmail.com> <1510157540.199886.1165880144.4BC2A083@webmail.messagingengine.com> Message-ID: <1510160072.928741.1165937696.5F4EDE1F@webmail.messagingengine.com> But if it ignores it won't you just not get your files in the tarball? -- Tristan Sloughter "I am not a crackpot" - Abe Simpson t@REDACTED On Wed, Nov 8, 2017, at 08:46 AM, Alexander Petrovsky wrote: > > > 2017-11-08 19:12 GMT+03:00 Tristan Sloughter : >> __ >> So the issue is simply the use of copying to `./` as opposed to >> `./somedir`, right?> > Exactly! > >> >> >> Certainly an issue, easy fix is to put it under another diectory, >> since you can't access it with `code:priv_dir` anyway since it isn't >> under an application no reason not to do `{copy, "priv", "etc/"}` but >> you probably already did that.>> >> I'd like to see the fix, if possible, in erl_tar but I'd have to take >> a closer look to know for sure if that is possible and what it would >> be. Not sure when I'll get time to dig in.> > I'm agree that it's will be perfect to fix the problem in erl_tar, but > as quick fix we can ignore ./ mapping in > https://github.com/erlware/relx/blob/master/src/rlx_prv_archive.erl#L111, > and teach overlay_files ignore ./ mappings> >> >> >> -- >> Tristan Sloughter >> "I am not a crackpot" - Abe Simpson >> t@REDACTED >> >> >> On Wed, Nov 8, 2017, at 07:52 AM, Alexander Petrovsky wrote: >>> >>> >>> 2017-11-08 16:25 GMT+03:00 Kostis Sagonas : >>>> On 11/08/2017 11:36 AM, Alexander Petrovsky wrote: >>>>> Would be great to hear rebar3 maintainers opinion.>>>> >>>> Why don't you send this as an issue in rebar3's repo then? >>>> >>>> Kostis>>> >>> It's not only rebar3 problem. If you read the letter, you should >>> notice, the problem occur in erl_tar.>>> >>> >>> -- >>> ?????????? ????????? / Alexander Petrovsky, >>> >>> Skype: askjuise >>> Phone: +7 931 9877991 >>> >>> >>> _________________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 931 9877991 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From facundo.olano@REDACTED Wed Nov 8 23:36:05 2017 From: facundo.olano@REDACTED (Facundo Olano) Date: Wed, 8 Nov 2017 19:36:05 -0300 Subject: [erlang-questions] Refreshing ssl certificates in a cowboy application In-Reply-To: References: Message-ID: I just tested this and indeed it works by default. Thanks! On Wed, Nov 8, 2017 at 12:54 PM, Danil Zagoskin wrote: > Hi! > > OTP's ssl subsystem with default settings reloads the certfile without any > extra actions. > See ssl_pem_cache.erl for implementation details. > > On Wed, Nov 8, 2017 at 5:44 PM, Facundo Olano < > facundo.olano@REDACTED> wrote: > >> Hi! >> >> We are serving a cowboy web application with certbot/let's encrypt >> certificates which expire every 90 days. Since the cert files are changing >> so often, I'm wondering if there's a way to reload them without having to >> restart the application. >> >> I understand cowboy ultimately uses the Erlang ssl module, so I guess >> this is not a cowboy specific issue. >> >> Thanks, >> Facundo. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > Danil Zagoskin | z@REDACTED > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Nov 9 00:00:01 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 8 Nov 2017 23:00:01 +0000 Subject: [erlang-questions] Refreshing ssl certificates in a cowboy application In-Reply-To: References: Message-ID: <81abaace-7a3b-8b93-4bc5-f60da97f9465@ninenines.eu> Oh good to know, I had no idea. Thanks for the tip. On 11/08/2017 03:54 PM, Danil Zagoskin wrote: > Hi! > > OTP's ssl subsystem with default settings reloads the certfile without > any extra actions. > See?ssl_pem_cache.erl for implementation details. > > On Wed, Nov 8, 2017 at 5:44 PM, Facundo Olano > > > wrote: > > Hi! > > We are serving a cowboy web application with certbot/let's encrypt > certificates which expire every 90 days. Since the cert files are > changing so often, I'm wondering if there's a way to reload them > without having to restart the application. > > I understand cowboy ultimately uses the Erlang ssl module, so I > guess this is not a cowboy specific issue. > > Thanks, > Facundo. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > -- > Danil Zagoskin | z@REDACTED > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From skypexu@REDACTED Thu Nov 9 04:06:25 2017 From: skypexu@REDACTED (Skype Xu) Date: Thu, 9 Nov 2017 11:06:25 +0800 Subject: [erlang-questions] weak resource reference possible in nif ? In-Reply-To: <1510138763.25410.24.camel@ericsson.com> References: <0348cae8-b460-ef7b-48a9-f35de4c863fd@gmail.com> <1510138763.25410.24.camel@ericsson.com> Message-ID: <877d6914-2e75-f25d-d769-842762b6e5a3@gmail.com> We don't want to get a new resource term every time when we got a new notification, we want to simply use erlang pattern matching without another function to retrieve actual data in the resource and matching against the data,? if we do this,? a programmer can keep the data around for later matching, and may forget to keep the resource, then what's the reason to use erlang resource? we would give up. On 2017?11?08? 18:59, John H?gberg wrote: > Hi, > > Why not let the resource hold nothing besides a shared_ptr to the > actual data, and use weak_ptrs in the internal list? > > You'd need to allocate a new resource when sending notifications from > that internal thread (Since the original resource could have been > destroyed), but other than that it should be pretty straightforward > unless I've misunderstood something. > > Regards, > John H?gberg > > On ons, 2017-11-08 at 17:32 +0800, Skype Xu wrote: >> Hi, >> >> Suppose a nif module is multi-threaded, we keep a pointer to a >> resource >> in the >> module's internal list, when the resource is no longer used by >> erlang >> process, >> erlang vm calls the resource's destructor, and we remove the pointer >> from >> internal list in the destructor, that's fine. >> However a problem arises when the internal list is also used by >> another >> thread >> created in the nif module, when some external events were triggered, >> the >> thread >> tries to use the resource pointer and sends it to a process in the >> vm, >> problem >> is there's a race condition between destructor and the internal >> thread, >> in the destructor we try to remove it from the internal list, at the >> time, the >> resource's reference count is already zero, and before the destructor >> is >> called, >> the internal thread sent the resource to a proecess, by calling >> enif_make_resource() to increase the count to 1, but anyway after the >> destructor returned, the vm will recycle the memory of the resource >> despites >> the reference count is 1 again, this leads to corrupted memory. >> >> We can not keep a resource by adding extra reference count in nif >> module, >> if we do, the resource's destructor will never be called even if not >> any >> erlang >> process is using the resource, because the reference count will never >> to >> dropped to 0. >> >> It would be perfect if we can keep a weak reference in nif like the >> one in >> C++ does, one can use weak_ptr to get back a shared_ptr object if >> the >> object >> is still using, otherwise a null pointer is returned.? if this is >> implemented, we >> can queue a notification to the internal thread and the thread will >> remove >> any data related to the resource.? enif_monitor_process() is not a >> solution >> because we want vm to recycle the resource when it is not used, not >> after >> a process crashed or exited. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From max.lapshin@REDACTED Thu Nov 9 07:31:22 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 9 Nov 2017 09:31:22 +0300 Subject: [erlang-questions] Refreshing ssl certificates in a cowboy application In-Reply-To: <81abaace-7a3b-8b93-4bc5-f60da97f9465@ninenines.eu> References: <81abaace-7a3b-8b93-4bc5-f60da97f9465@ninenines.eu> Message-ID: we use it for automatic refreshing letsencrypt certificates. Works good. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fxn@REDACTED Thu Nov 9 09:27:05 2017 From: fxn@REDACTED (Xavier Noria) Date: Thu, 9 Nov 2017 09:27:05 +0100 Subject: [erlang-questions] uses cases for limiting schedulers or threads? Message-ID: I read in Cesarini's book that you can limit the number of schedulers and threads (guess by "threads" it means the size of the thread pool for I/O). Have you ever needed this? If so, which was the use case? -------------- next part -------------- An HTML attachment was scrubbed... URL: From fxn@REDACTED Thu Nov 9 09:28:14 2017 From: fxn@REDACTED (Xavier Noria) Date: Thu, 9 Nov 2017 09:28:14 +0100 Subject: [erlang-questions] uses cases for limiting schedulers or threads? In-Reply-To: References: Message-ID: Sorry, Cesarini & Vinoski. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Nov 9 13:05:06 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 9 Nov 2017 15:05:06 +0300 Subject: [erlang-questions] uses cases for limiting schedulers or threads? In-Reply-To: References: Message-ID: For example, you want to run redis/tarantool and erlang on the same machine. It may be a good idea to limit amount of schedulers to N-1 cores. Or you have some task once per hour that consumes full core, better leave erlang without some cores. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Nov 9 14:45:55 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 9 Nov 2017 13:45:55 +0000 Subject: [erlang-questions] [ANN] Cowboy 2.1 Message-ID: Cowboy 2.1.0 has been released! This release focuses on adding features that were temporarily removed during the 2.0 release process, fixing a few bugs at the same time. Please see the complete announcement at: https://ninenines.eu/articles/cowboy-2.1.0/ Cheers, -- Lo?c Hoguin https://ninenines.eu From eric.des.courtis@REDACTED Thu Nov 9 16:17:01 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Thu, 9 Nov 2017 10:17:01 -0500 Subject: [erlang-questions] Refreshing ssl certificates in a cowboy application In-Reply-To: References: <81abaace-7a3b-8b93-4bc5-f60da97f9465@ninenines.eu> Message-ID: That is excellent. We have a situation where we have customers that bring their own domain. Is it possible to add an SSL cert in flight? On Thu, Nov 9, 2017 at 1:31 AM, Max Lapshin wrote: > we use it for automatic refreshing letsencrypt certificates. Works good. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivancarmenates@REDACTED Thu Nov 9 19:17:18 2017 From: ivancarmenates@REDACTED (=?UTF-8?Q?Iv=C3=A1n_Carmenates?=) Date: Thu, 9 Nov 2017 13:17:18 -0500 Subject: [erlang-questions] Fwd: About a multicore benchmark In-Reply-To: References: Message-ID: Hi, I am trying to do a simple benchmark against python, and we decide to use factorial function to the test fact(0)->1; fact(N)->N*fact(N-1). But I wonder if there is a way to implement this using a multicore approach. Kindest regards, Ivan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Fri Nov 10 08:08:59 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 10 Nov 2017 09:08:59 +0200 Subject: [erlang-questions] Fwd: About a multicore benchmark In-Reply-To: References: Message-ID: Hello, The parallel implementation of Factorial algorithm has been studies for a while. http://swvist.github.io/09-13-2012-naive_parallel_factorial_in_erlang.html https://stackoverflow.com/questions/34978236/parallelizing-factorial-calculation This algorithm would not give you a representative benchmark vs Python. I am afraid that any math problem would be weaker in Erlang unless you implement it as NIF. You need to design benchmark strategy according to your use-cases. Best Regards, Dmitry > On 9 Nov 2017, at 20.17, Iv?n Carmenates wrote: > > Hi, I am trying to do a simple benchmark against python, and we decide to use factorial function to the test > > fact(0)->1; > fact(N)->N*fact(N-1). > > But I wonder if there is a way to implement this using a multicore approach. > > Kindest regards, > Ivan. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hogberg@REDACTED Fri Nov 10 11:27:01 2017 From: john.hogberg@REDACTED (=?utf-8?B?Sm9obiBIw7ZnYmVyZw==?=) Date: Fri, 10 Nov 2017 10:27:01 +0000 Subject: [erlang-questions] weak resource reference possible in nif ? In-Reply-To: <877d6914-2e75-f25d-d769-842762b6e5a3@gmail.com> References: <0348cae8-b460-ef7b-48a9-f35de4c863fd@gmail.com> <1510138763.25410.24.camel@ericsson.com> <877d6914-2e75-f25d-d769-842762b6e5a3@gmail.com> Message-ID: <1510309621.25410.185.camel@ericsson.com> I just thought of a dirty hack that might work; serialize the resource on creation with enif_term_to_binary and store *that* in your internal list instead of the resource, then send the result of enif_binary_to_term in each notification. The resource won't be kept alive any longer than needed (As the binary won't keep any references to it), and it won't crash if the resource has been deallocated as enif_binary_to_term will recognize it as such and return a stale resource. It will only work on OTP 20 or later though. http://erlang.org/doc/man/erl_nif.html#enif_make_resource /John On tor, 2017-11-09 at 11:06 +0800, Skype Xu wrote: > We don't want to get a new resource term every time when we got a > new? > notification, > we want to simply use erlang pattern matching without another > function? > to retrieve > actual data in the resource and matching against the data,? if we do? > this,? a programmer > can keep the data around for later matching, and may forget to keep > the? > resource, then > what's the reason to use erlang resource? we would give up. From 249505968@REDACTED Fri Nov 10 11:08:39 2017 From: 249505968@REDACTED (=?gb18030?B?99L30Q==?=) Date: Fri, 10 Nov 2017 18:08:39 +0800 Subject: [erlang-questions] how to get a path in code with rebar3 umbrella project Message-ID: I build a project use rebar3 new release add two app in it. and in app A, i need get the another app B's ebin directory in code how to configure this with rebar3? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gordeev.vladimir.v@REDACTED Fri Nov 10 13:38:31 2017 From: gordeev.vladimir.v@REDACTED (Vladimir Gordeev) Date: Fri, 10 Nov 2017 15:38:31 +0300 Subject: [erlang-questions] how to get a path in code with rebar3 umbrella project In-Reply-To: References: Message-ID: Hey! Maybe this will work for you: ./rebar3 path --app myapp1,myapp2 You can read more from ./rebar3 help path On Fri, Nov 10, 2017 at 1:08 PM, ?? <249505968@REDACTED> wrote: > I build a project use rebar3 new release > add two app in it. > and in app A, i need get the another app B's ebin directory in code > how to configure this with rebar3? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Fri Nov 10 14:47:29 2017 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Fri, 10 Nov 2017 13:47:29 +0000 Subject: [erlang-questions] how to get a path in code with rebar3 umbrella project In-Reply-To: References: Message-ID: Have you tried code:lib_dir(appB)? On Fri 10 Nov 2017 at 13:41, Vladimir Gordeev wrote: > Hey! > > Maybe this will work for you: ./rebar3 path --app myapp1,myapp2 > You can read more from ./rebar3 help path > > On Fri, Nov 10, 2017 at 1:08 PM, ?? <249505968@REDACTED> wrote: > >> I build a project use rebar3 new release >> add two app in it. >> and in app A, i need get the another app B's ebin directory in code >> how to configure this with rebar3? >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Cheers, -- Pierre Fenoll -------------- next part -------------- An HTML attachment was scrubbed... URL: From ates@REDACTED Fri Nov 10 16:15:18 2017 From: ates@REDACTED (Artem Teslenko) Date: Fri, 10 Nov 2017 17:15:18 +0200 Subject: [erlang-questions] No SCTP support in ESL Erlang Message-ID: <3dfe6682-96b3-ecbe-61c1-2b23580f4782@ipv6.dp.ua> Hello, Looks like the latest release of the Erlang from ESL is compiled without SCTP support: OS: $ cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) Erlang: $ rpm -qa |grep erlang esl-erlang-20.1-1.x86_64 lksctp-tools rpm is installed and sctp kernel module is loaded too. Output from the shell: $ erl Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:56:56] [ds:56:56:10] [async-threads:10] [hipe] [kernel-poll:false] Eshell V9.1? (abort with ^G) 1> gen_sctp:open(). {error,eprotonosupport} Is there anyone from ESL who could check this? Thank you. From ingela.anderton.andin@REDACTED Fri Nov 10 16:51:08 2017 From: ingela.anderton.andin@REDACTED (Ingela Anderton Andin) Date: Fri, 10 Nov 2017 16:51:08 +0100 Subject: [erlang-questions] Patch package OTP 20.1.5 released Message-ID: <20171110155108.GA58005@duper.otp.ericsson.se> Patch Package: OTP 20.1.5 Git Tag: OTP-20.1.5 Date: 2017-11-10 Trouble Report Id: OTP-14707, OTP-14730, OTP-14745, OTP-14750, OTP-14751, OTP-14761 Seq num: ERL-507 System: OTP Release: 20 Application: erts-9.1.4, inets-6.4.4 Predecessor: OTP 20.1.4 Check out the git tag OTP-20.1.5, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- erts-9.1.4 ------------------------------------------------------ --------------------------------------------------------------------- The erts-9.1.4 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-14707 Application(s): erts Microstate accounting sometimes produced incorrect results for dirty schedulers. OTP-14730 Application(s): erts Related Id(s): ERL-507 Fixed a regression in zlib:gunzip/1 that prevented it from working when the decompressed size was a perfect multiple of 16384. This regression was introduced in 20.1.1 OTP-14745 Application(s): erts Fixed a memory corruption bug in enif_inspect_iovec; writable binaries stayed writable after entering the iovec. OTP-14750 Application(s): erts Fixed a crash in enif_inspect_iovec on encountering empty binaries. OTP-14751 Application(s): erts zlib:deflateParams/3 will no longer return buf_error when called after zlib:deflate/2 with zlib 1.2.11. Full runtime dependencies of erts-9.1.4: kernel-5.0, sasl-3.0.1, stdlib-3.0 --------------------------------------------------------------------- --- inets-6.4.4 ----------------------------------------------------- --------------------------------------------------------------------- The inets-6.4.4 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-14761 Application(s): inets Correct the handling of location headers so that the status code is not hard coded. This should have been fixed by commit 2cc5ba70cbbc6b3ace81a2a0324417c3b65265bb but unfortunately was broken during a code refactoring and unnoticed due to a faulty placed test case. Full runtime dependencies of inets-6.4.4: erts-6.0, kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, ssl-5.3.4, stdlib-2.0 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From eshikafe@REDACTED Sun Nov 12 09:54:45 2017 From: eshikafe@REDACTED (austin aigbe) Date: Sun, 12 Nov 2017 09:54:45 +0100 Subject: [erlang-questions] Patch Package OTP 20.1.3 Released In-Reply-To: <1508915516.2025.5.camel@ericsson.com> References: <1508915516.2025.5.camel@ericsson.com> Message-ID: Thanks John. How do I install diameter-2.1 on Windows without installing from source? Previous versions come with all the modules precompiled. I just noticed that the diameter beam files are not included in OTP 20.1. I keep getting the following error after upgrading from OTP 19.3 to OTP 20.1 11> ls(). cx_ts29229_v1270.dia ok 12> diameter_make:codec("cx_ts29229_v1270.dia", [erl]). ** exception error: undefined function diameter_make:codec/2 13> Is installing from source the only option? [image: Inline image 1] On Wed, Oct 25, 2017 at 8:11 AM, John H?gberg wrote: > Patch Package: OTP 20.1.3 > Git Tag: OTP-20.1.3 > Date: 2017-10-25 > Trouble Report Id: OTP-13372, OTP-14672, OTP-14684, OTP-14694 > Seq num: ERIERL-73, ERIERL-85, ERL-494 > System: OTP > Release: 20 > Application: diameter-2.1.2, erts-9.1.3, snmp-5.2.8 > Predecessor: OTP 20.1.2 > > Check out the git tag OTP-20.1.3, and build a full OTP system > including documentation. Apply one or more applications from this > build as patches to your installation using the 'otp_patch_apply' > tool. For information on install requirements, see descriptions for > each application version below. > > --------------------------------------------------------------------- > --- diameter-2.1.2 -------------------------------------------------- > --------------------------------------------------------------------- > > The diameter-2.1.2 application can be applied independently of other > applications on a full OTP 20 installation. > > --- Fixed Bugs and Malfunctions --- > > OTP-14684 Application(s): diameter > Related Id(s): ERIERL-85 > > A fault introduced in diameter 2.1 could cause decode > errors to be ignored in AVPs following the header of a > Grouped AVP. > > > Full runtime dependencies of diameter-2.1.2: erts-6.4, kernel-3.2, > ssl-6.0, stdlib-2.4 > > > --------------------------------------------------------------------- > --- erts-9.1.3 ------------------------------------------------------ > --------------------------------------------------------------------- > > The erts-9.1.3 application can be applied independently of other > applications on a full OTP 20 installation. > > --- Fixed Bugs and Malfunctions --- > > OTP-14672 Application(s): erts > Related Id(s): ERL-494 > > Added zlib:set_controlling_process/2 to move a > zstream() between processes. > > > OTP-14694 Application(s): erts > > Fix so that schedulers are bound correctly when the > first available cpu is not the first detected cpu. e.g. > when using "taskset -c X..Y" when X is not equal to 0. > > > Full runtime dependencies of erts-9.1.3: kernel-5.0, sasl-3.0.1, > stdlib-3.0 > > > --------------------------------------------------------------------- > --- snmp-5.2.8 ------------------------------------------------------ > --------------------------------------------------------------------- > > The snmp-5.2.8 application can be applied independently of other > applications on a full OTP 20 installation. > > --- Fixed Bugs and Malfunctions --- > > OTP-13372 Application(s): snmp > Related Id(s): ERIERL-73 > > The recbuf configuration option was not propagated > correctly to the socket for the SNMP Manager. > > > Full runtime dependencies of snmp-5.2.8: crypto-3.3, erts-6.0, > kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, stdlib-2.5 > > > --------------------------------------------------------------------- > --------------------------------------------------------------------- > --------------------------------------------------------------------- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 23225 bytes Desc: not available URL: From eshikafe@REDACTED Sun Nov 12 10:20:27 2017 From: eshikafe@REDACTED (austin aigbe) Date: Sun, 12 Nov 2017 10:20:27 +0100 Subject: [erlang-questions] Patch Package OTP 20.1.3 Released In-Reply-To: References: <1508915516.2025.5.camel@ericsson.com> Message-ID: I upgraded from the following version: Erlang/OTP 20 [erts-9.0] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] Eshell V9.0 (abort with ^G) On Sun, Nov 12, 2017 at 9:54 AM, austin aigbe wrote: > Thanks John. > > How do I install diameter-2.1 on Windows without installing from source? > Previous versions come with all the modules precompiled. > > I just noticed that the diameter beam files are not included in OTP 20.1. > > I keep getting the following error after upgrading from OTP 19.3 to OTP > 20.1 > > 11> ls(). > cx_ts29229_v1270.dia > ok > 12> diameter_make:codec("cx_ts29229_v1270.dia", [erl]). > ** exception error: undefined function diameter_make:codec/2 > 13> > > Is installing from source the only option? > > [image: Inline image 1] > > > > On Wed, Oct 25, 2017 at 8:11 AM, John H?gberg > wrote: > >> Patch Package: OTP 20.1.3 >> Git Tag: OTP-20.1.3 >> Date: 2017-10-25 >> Trouble Report Id: OTP-13372, OTP-14672, OTP-14684, OTP-14694 >> Seq num: ERIERL-73, ERIERL-85, ERL-494 >> System: OTP >> Release: 20 >> Application: diameter-2.1.2, erts-9.1.3, snmp-5.2.8 >> Predecessor: OTP 20.1.2 >> >> Check out the git tag OTP-20.1.3, and build a full OTP system >> including documentation. Apply one or more applications from this >> build as patches to your installation using the 'otp_patch_apply' >> tool. For information on install requirements, see descriptions for >> each application version below. >> >> --------------------------------------------------------------------- >> --- diameter-2.1.2 -------------------------------------------------- >> --------------------------------------------------------------------- >> >> The diameter-2.1.2 application can be applied independently of other >> applications on a full OTP 20 installation. >> >> --- Fixed Bugs and Malfunctions --- >> >> OTP-14684 Application(s): diameter >> Related Id(s): ERIERL-85 >> >> A fault introduced in diameter 2.1 could cause decode >> errors to be ignored in AVPs following the header of a >> Grouped AVP. >> >> >> Full runtime dependencies of diameter-2.1.2: erts-6.4, kernel-3.2, >> ssl-6.0, stdlib-2.4 >> >> >> --------------------------------------------------------------------- >> --- erts-9.1.3 ------------------------------------------------------ >> --------------------------------------------------------------------- >> >> The erts-9.1.3 application can be applied independently of other >> applications on a full OTP 20 installation. >> >> --- Fixed Bugs and Malfunctions --- >> >> OTP-14672 Application(s): erts >> Related Id(s): ERL-494 >> >> Added zlib:set_controlling_process/2 to move a >> zstream() between processes. >> >> >> OTP-14694 Application(s): erts >> >> Fix so that schedulers are bound correctly when the >> first available cpu is not the first detected cpu. e.g. >> when using "taskset -c X..Y" when X is not equal to 0. >> >> >> Full runtime dependencies of erts-9.1.3: kernel-5.0, sasl-3.0.1, >> stdlib-3.0 >> >> >> --------------------------------------------------------------------- >> --- snmp-5.2.8 ------------------------------------------------------ >> --------------------------------------------------------------------- >> >> The snmp-5.2.8 application can be applied independently of other >> applications on a full OTP 20 installation. >> >> --- Fixed Bugs and Malfunctions --- >> >> OTP-13372 Application(s): snmp >> Related Id(s): ERIERL-73 >> >> The recbuf configuration option was not propagated >> correctly to the socket for the SNMP Manager. >> >> >> Full runtime dependencies of snmp-5.2.8: crypto-3.3, erts-6.0, >> kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, stdlib-2.5 >> >> >> --------------------------------------------------------------------- >> --------------------------------------------------------------------- >> --------------------------------------------------------------------- >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 23225 bytes Desc: not available URL: From frank.muller.erl@REDACTED Sun Nov 12 12:50:58 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sun, 12 Nov 2017 11:50:58 +0000 Subject: [erlang-questions] beam.smp high memory usage In-Reply-To: References: Message-ID: Hi guys, I?ve a fairly complex I/O bound Erlang application. Since yesterday, the beam.smp memory usage dramatically increased from ~160MB to something between [2.5GB, 3.1GB]. I?ve no idea how, nor why :-/ Attached is snapshot taken with recon_alloc:snapshot(). Please let me know if you want anything else. Hope someone can explain me what?s going on. Thank you. /Frank INFO: . Physical machine (not a VM) . Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] . CentOS-7-x86_64 . Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux . 8x Intel(R) Xeon(R) CPU E5504 @ 2.00GHz . 16GB of RAM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: recon_snapshot.terms Type: application/octet-stream Size: 129835 bytes Desc: not available URL: From ok@REDACTED Sun Nov 12 13:44:27 2017 From: ok@REDACTED (ok@REDACTED) Date: Mon, 13 Nov 2017 01:44:27 +1300 Subject: [erlang-questions] Fwd: About a multicore benchmark In-Reply-To: References: Message-ID: <235b9109af685630dd9ef4dab6dff838.squirrel@chasm.otago.ac.nz> > Hi, I am trying to do a simple benchmark against python, and we decide to > use factorial function to the test > > fact(0)->1; > fact(N)->N*fact(N-1). > > But I wonder if there is a way to implement this using a multicore > approach. Fairly obviously, yes. Suppose you have k cores. Core 1 computes 1 * (k+1) * (2k+1) ... Core 2 computes 2 * (k+2) * (2k+2) ... Core 3 computes 3 * (k+3) * (2k+3) ... ... Core k computes k * (k+k) * (2k+k) ... and finally you take the product of their answers. The problem is that it is difficult to see this benchmark as telling you anything worth knowing. Factorials get very big very fast. So for any N big enough to be worth measuring, what you are really measuring is the speed of the under-lying bignum library. I expect that Erlang and Python use the *same* bignum library (gmp), so what does that tell you about anything *else* in Erlang and Python? For what it's worth, here's your benchmark in Smalltalk: Time millisecondsToRun: [20000 factorial] System A: 355 msec. System B: 347 msec (noise level difference). System C: 189 msec. What does this actually tell us about these systems? Very little. It just so happens that System C (which is mine) has a carefully written #factorial function which runs, in System A or System B, about 40% faster than the naive code they have. The underlying bignum library in System C is actually significantly *worse* than the ones in Systems A and B for large enough products. If you want some marginally less unrealistic benchmarks, you could look at "The Computer Languages Benchmarks Game" http://benchmarksgame.alioth.debian.org/ You should certainly read http://benchmarksgame.alioth.debian.org/why-measure-toy-benchmark-programs.html The Chameneos-Redux benchmark has Erlang at 62.3 seconds and Python3 at 236.84 seconds. However, when I tried my own implementation in Smalltalk, it turned out that almost all the time was going into shared queue operations and a small tweak to shared queues *doubled* the speed of the program. If for example you turn to the NBody benchmark, what you mostly learn is that programming languages that use "boxed" floating point numbers do much worse than languages with unboxed ones, unless they have exceedingly clever compilers. Erlang's performance suffers for that reason, but Python does worse. Here again, a relatively simple compiler hack (turning x := t * vx + x into if t, vx, and x are all FloatD, then BOXD(UNBOXD(t)*UNBOXD(vx)+UNBOXD(x) else ...) can halve the amount of boxing, thus doubling the speed. This is something I believe HiPE can do, given suitable type information. The lesson is that small benchmarks can be extremely sensitive to implementation details that are much less unfluential for large scale systems. Benchmarking is a wonderful way to ask "where is the time going? how can I do better?" but a lousy way to compare languages. From jesper.louis.andersen@REDACTED Sun Nov 12 16:35:16 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 12 Nov 2017 15:35:16 +0000 Subject: [erlang-questions] beam.smp high memory usage In-Reply-To: References: Message-ID: First quick stab: Most of the allocated space is binary() data. Typical scenario: something is keeping the binary() data alive, one way or the other. Either because you need a binary:copy/1 in key locations due to sub-binaries, or because your system can't get rid of the data since you still have a pointer kept to it from some process. On Sun, Nov 12, 2017 at 12:51 PM Frank Muller wrote: > Hi guys, > > I?ve a fairly complex I/O bound Erlang application. > > Since yesterday, the beam.smp memory usage dramatically increased from > ~160MB to something between [2.5GB, 3.1GB]. > I?ve no idea how, nor why :-/ > > Attached is snapshot taken with recon_alloc:snapshot(). > Please let me know if you want anything else. > > Hope someone can explain me what?s going on. > > Thank you. > /Frank > > INFO: > . Physical machine (not a VM) > . Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] > [async-threads:10] [hipe] [kernel-poll:false] > . CentOS-7-x86_64 > . Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST > 2017 x86_64 x86_64 x86_64 GNU/Linux > . 8x Intel(R) Xeon(R) CPU E5504 @ 2.00GHz > . 16GB of RAM > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sun Nov 12 23:36:03 2017 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Sun, 12 Nov 2017 23:36:03 +0100 Subject: [erlang-questions] [ANN] geas 2.0.13 Message-ID: <8721febf-2b27-d734-b177-053a04c4002f@wanadoo.fr> Hi, Geas 2.0.13 has been released, with some delay sorry ! This is an update for Erlang 20.1 database. Geas is a tool that will detect the runnable official Erlang release window for your project, including dependencies, either from source code or beam files. Geas will tell you what are the offending functions in the beam/source files that reduce the available window. Geas will tell you if some beam files are compiled native. Geas is available as a module, erlang.mk and rebar 2/3 plugins. https://github.com/crownedgrouse/geas Cheers ! Eric From frank.muller.erl@REDACTED Mon Nov 13 09:23:39 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 13 Nov 2017 08:23:39 +0000 Subject: [erlang-questions] beam.smp high memory usage In-Reply-To: References: Message-ID: Hi Jesper, Thanks again for the feedback. But from where should I start (Erlang newbie here)? I?ve no idea ... is there a guideline to follow? I feel guessing without making any real progress for the last two days :-/ /Frank First quick stab: > > Most of the allocated space is binary() data. > > Typical scenario: something is keeping the binary() data alive, one way or > the other. Either because you need a binary:copy/1 in key locations due to > sub-binaries, or because your system can't get rid of the data since you > still have a pointer kept to it from some process. > > > On Sun, Nov 12, 2017 at 12:51 PM Frank Muller > wrote: > >> Hi guys, >> >> I?ve a fairly complex I/O bound Erlang application. >> >> Since yesterday, the beam.smp memory usage dramatically increased from >> ~160MB to something between [2.5GB, 3.1GB]. >> I?ve no idea how, nor why :-/ >> >> Attached is snapshot taken with recon_alloc:snapshot(). >> Please let me know if you want anything else. >> >> Hope someone can explain me what?s going on. >> >> Thank you. >> /Frank >> >> INFO: >> . Physical machine (not a VM) >> . Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] >> [async-threads:10] [hipe] [kernel-poll:false] >> . CentOS-7-x86_64 >> . Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST >> 2017 x86_64 x86_64 x86_64 GNU/Linux >> . 8x Intel(R) Xeon(R) CPU E5504 @ 2.00GHz >> . 16GB of RAM >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Nov 13 11:55:50 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 13 Nov 2017 10:55:50 +0000 Subject: [erlang-questions] beam.smp high memory usage In-Reply-To: References: Message-ID: Fred's excellent "Erlang In Anger" book (https://www.erlang-in-anger.com/) has a section on hunting down binary leaks (if that is what you are working with here). At least this can give you some starting information about what to look for and how to go about handling it afterwards. Identifying which processes are holding the binary data by extension identifies the code modules which are culprits. Since your normal operation uses 160 megabytes of memory and it suddenly shoots to gigabytes, one of the parts is to identify what is being done to the system when that happens. Perhaps you have an outside user which uses your system in a way that wasn't thought about originally. This can lead to you handling different data and in turn this can expose some of these problems. On Mon, Nov 13, 2017 at 9:23 AM Frank Muller wrote: > Hi Jesper, > > Thanks again for the feedback. > > But from where should I start (Erlang newbie here)? > I?ve no idea ... is there a guideline to follow? > > I feel guessing without making any real progress for the last two days :-/ > > /Frank > > First quick stab: >> >> Most of the allocated space is binary() data. >> >> Typical scenario: something is keeping the binary() data alive, one way >> or the other. Either because you need a binary:copy/1 in key locations due >> to sub-binaries, or because your system can't get rid of the data since you >> still have a pointer kept to it from some process. >> >> >> On Sun, Nov 12, 2017 at 12:51 PM Frank Muller >> wrote: >> >>> Hi guys, >>> >>> I?ve a fairly complex I/O bound Erlang application. >>> >>> Since yesterday, the beam.smp memory usage dramatically increased from >>> ~160MB to something between [2.5GB, 3.1GB]. >>> I?ve no idea how, nor why :-/ >>> >>> Attached is snapshot taken with recon_alloc:snapshot(). >>> Please let me know if you want anything else. >>> >>> Hope someone can explain me what?s going on. >>> >>> Thank you. >>> /Frank >>> >>> INFO: >>> . Physical machine (not a VM) >>> . Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] >>> [async-threads:10] [hipe] [kernel-poll:false] >>> . CentOS-7-x86_64 >>> . Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST >>> 2017 x86_64 x86_64 x86_64 GNU/Linux >>> . 8x Intel(R) Xeon(R) CPU E5504 @ 2.00GHz >>> . 16GB of RAM >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.sumner@REDACTED Mon Nov 13 22:01:07 2017 From: martin.sumner@REDACTED (Martin Sumner) Date: Mon, 13 Nov 2017 21:01:07 +0000 Subject: [erlang-questions] High Sierra and binary_to_term (compressed) Message-ID: Rather foolishly I accepted the upgrade to High Sierra on my Macbook last night. This has led to an interesting problem with a binary_to_term line on my project. Following the upgrade, tests have started failing on this line: https://github.com/martinsumner/leveled/blob/master/src/leveled_sst.erl#L875 with error:badarg being thrown. However if I copy and paste the binary on which the badarg has been thrown out of the crash report, and into a erlang console - binary_to_term works just fine. See this issue for an example - https://github.com/martinsumner/leveled/issues/106 This failure is intermittent, and indeed wrapping the binary_to_term like this is enough to make the problem seemingly disappear: try binary_to_term(Bin) of T -> T catch error:badarg -> binary_to_term(Bin) end. Re-running the tests on another OSX device on a release previous to High Sierra shows no issues with/without the wrapper. The problem appears to consistently exist in OTP 17.5 and 18.3. However, running the tests in 19.3 and R16B02-basho10 doesn't show the problem. The terms in question have all been compressed using term_to_binary's zlib compression. If I build 18.3 on High Sierra with the "--enable-builtin-zlib" option, the problem does *not* re-occur. This switch appears to be a reliable delta between having and not having the problem, and I believe may have become the default in OTP 17. It appears to be specific to v17/v18 Erlang with compressed terms, on High Sierra where the OS zlib implementation is used, not the inbuilt OTP zlib. There appears to be internet chatter of other issues with Mac OSX High Sierra and changes to the zlib implementation - https://github.com/madler/zlib/issues/305. So I think this doesn't look like it is an Erlang issue, but I suspect it may not be resolved in High Sierra any time soon, and may impact other Erlang users using zlib functionality. Does this it seem reasonable to pass this off as a High Sierra zlib bug, and use the --enable-builtin-zlib workaround? If this is the case I'm not sure why I don't get the issue on 19.3, any ideas as to why? Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jan.chochol@REDACTED Tue Nov 14 08:30:30 2017 From: jan.chochol@REDACTED (Jan Chochol) Date: Tue, 14 Nov 2017 08:30:30 +0100 Subject: [erlang-questions] High Sierra and binary_to_term (compressed) In-Reply-To: References: Message-ID: Hi Martin, This is bug with combination of new zlib and old Erlang - https://github.com/erlang/otp/commit/e27119948fc6ab28bea81019720bddaac5b655a7 . It is fixed in OTP-18.3.4.5 and newer. "--enable-builtin-zlib" works, because it uses older bundled zlib. Jan On Mon, Nov 13, 2017 at 10:01 PM, Martin Sumner wrote: > Rather foolishly I accepted the upgrade to High Sierra on my Macbook last > night. This has led to an interesting problem with a binary_to_term line > on my project. > > Following the upgrade, tests have started failing on this line: > > https://github.com/martinsumner/leveled/blob/master/src/leveled_sst.erl# > L875 > > with error:badarg being thrown. > > However if I copy and paste the binary on which the badarg has been thrown > out of the crash report, and into a erlang console - binary_to_term works > just fine. > > See this issue for an example - https://github.com/ > martinsumner/leveled/issues/106 > > This failure is intermittent, and indeed wrapping the binary_to_term like > this is enough to make the problem seemingly disappear: > > try binary_to_term(Bin) of > T -> > T > catch > error:badarg -> > binary_to_term(Bin) > end. > > Re-running the tests on another OSX device on a release previous to High > Sierra shows no issues with/without the wrapper. > > The problem appears to consistently exist in OTP 17.5 and 18.3. However, > running the tests in 19.3 and R16B02-basho10 doesn't show the problem. > > The terms in question have all been compressed using term_to_binary's zlib > compression. > > If I build 18.3 on High Sierra with the "--enable-builtin-zlib" option, > the problem does *not* re-occur. This switch appears to be a reliable > delta between having and not having the problem, and I believe may have > become the default in OTP 17. > > It appears to be specific to v17/v18 Erlang with compressed terms, on High > Sierra where the OS zlib implementation is used, not the inbuilt OTP zlib. > > There appears to be internet chatter of other issues with Mac OSX High > Sierra and changes to the zlib implementation - https://github.com/madler/ > zlib/issues/305. > > So I think this doesn't look like it is an Erlang issue, but I suspect it > may not be resolved in High Sierra any time soon, and may impact other > Erlang users using zlib functionality. Does this it seem reasonable to > pass this off as a High Sierra zlib bug, and use the --enable-builtin-zlib > workaround? If this is the case I'm not sure why I don't get the issue on > 19.3, any ideas as to why? > > Martin > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From liuzhongzheng2012@REDACTED Tue Nov 14 08:56:17 2017 From: liuzhongzheng2012@REDACTED (Zhongzheng Liu) Date: Tue, 14 Nov 2017 15:56:17 +0800 Subject: [erlang-questions] Which built-in function makes [H|T] ? Message-ID: Hi mail-list: Which built-in function work as fun(H, T) -> [H|T] end ? i.e how to fill the following expression without user defined function ? [H|T] = erlang:apply(???, ???, [H, T]). Thanks Liu zhongzheng From skypexu@REDACTED Tue Nov 14 09:11:23 2017 From: skypexu@REDACTED (Skype Xu) Date: Tue, 14 Nov 2017 16:11:23 +0800 Subject: [erlang-questions] weak resource reference possible in nif ? In-Reply-To: <1510309621.25410.185.camel@ericsson.com> References: <0348cae8-b460-ef7b-48a9-f35de4c863fd@gmail.com> <1510138763.25410.24.camel@ericsson.com> <877d6914-2e75-f25d-d769-842762b6e5a3@gmail.com> <1510309621.25410.185.camel@ericsson.com> Message-ID: Yes, I have tried OTP 20, it works, also the dirty hack is too expensive to run. On 2017?11?10? 18:27, John H?gberg wrote: > I just thought of a dirty hack that might work; serialize the resource > on creation with enif_term_to_binary and store *that* in your internal > list instead of the resource, then send the result of > enif_binary_to_term in each notification. > > The resource won't be kept alive any longer than needed (As the binary > won't keep any references to it), and it won't crash if the resource > has been deallocated as enif_binary_to_term will recognize it as such > and return a stale resource. > > It will only work on OTP 20 or later though. > > http://erlang.org/doc/man/erl_nif.html#enif_make_resource > > /John > > On tor, 2017-11-09 at 11:06 +0800, Skype Xu wrote: >> We don't want to get a new resource term every time when we got a >> new >> notification, >> we want to simply use erlang pattern matching without another >> function >> to retrieve >> actual data in the resource and matching against the data,? if we do >> this,? a programmer >> can keep the data around for later matching, and may forget to keep >> the >> resource, then >> what's the reason to use erlang resource? we would give up. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From martin.sumner@REDACTED Tue Nov 14 09:09:29 2017 From: martin.sumner@REDACTED (Martin Sumner) Date: Tue, 14 Nov 2017 08:09:29 +0000 Subject: [erlang-questions] High Sierra and binary_to_term (compressed) In-Reply-To: References: Message-ID: Jan, Thank-you, that does explain things. Martin On 14 November 2017 at 07:30, Jan Chochol wrote: > Hi Martin, > > This is bug with combination of new zlib and old Erlang - > https://github.com/erlang/otp/commit/e27119948fc6ab28bea81019720bdd > aac5b655a7 . > It is fixed in OTP-18.3.4.5 and newer. > "--enable-builtin-zlib" works, because it uses older bundled zlib. > > Jan > > On Mon, Nov 13, 2017 at 10:01 PM, Martin Sumner > wrote: > >> Rather foolishly I accepted the upgrade to High Sierra on my Macbook last >> night. This has led to an interesting problem with a binary_to_term line >> on my project. >> >> Following the upgrade, tests have started failing on this line: >> >> https://github.com/martinsumner/leveled/blob/master/src/ >> leveled_sst.erl#L875 >> >> with error:badarg being thrown. >> >> However if I copy and paste the binary on which the badarg has been >> thrown out of the crash report, and into a erlang console - binary_to_term >> works just fine. >> >> See this issue for an example - https://github.com/martinsum >> ner/leveled/issues/106 >> >> This failure is intermittent, and indeed wrapping the binary_to_term like >> this is enough to make the problem seemingly disappear: >> >> try binary_to_term(Bin) of >> T -> >> T >> catch >> error:badarg -> >> binary_to_term(Bin) >> end. >> >> Re-running the tests on another OSX device on a release previous to High >> Sierra shows no issues with/without the wrapper. >> >> The problem appears to consistently exist in OTP 17.5 and 18.3. However, >> running the tests in 19.3 and R16B02-basho10 doesn't show the problem. >> >> The terms in question have all been compressed using term_to_binary's >> zlib compression. >> >> If I build 18.3 on High Sierra with the "--enable-builtin-zlib" option, >> the problem does *not* re-occur. This switch appears to be a reliable >> delta between having and not having the problem, and I believe may have >> become the default in OTP 17. >> >> It appears to be specific to v17/v18 Erlang with compressed terms, on >> High Sierra where the OS zlib implementation is used, not the inbuilt OTP >> zlib. >> >> There appears to be internet chatter of other issues with Mac OSX High >> Sierra and changes to the zlib implementation - >> https://github.com/madler/zlib/issues/305. >> >> So I think this doesn't look like it is an Erlang issue, but I suspect it >> may not be resolved in High Sierra any time soon, and may impact other >> Erlang users using zlib functionality. Does this it seem reasonable to >> pass this off as a High Sierra zlib bug, and use the --enable-builtin-zlib >> workaround? If this is the case I'm not sure why I don't get the issue on >> 19.3, any ideas as to why? >> >> Martin >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elinsn@REDACTED Tue Nov 14 09:32:38 2017 From: elinsn@REDACTED (Sergey Yelin) Date: Tue, 14 Nov 2017 11:32:38 +0300 Subject: [erlang-questions] High Sierra and binary_to_term (compressed) In-Reply-To: References: Message-ID: The `--enable-builtin-zlib` flag also fixes strange xref/dialyzer bugs in one of our projects built on top of Erlang 17.5 on High Sierra. Thanks for sharing! :) --- Best regards, Sergey Yelin. > On 14 Nov 2017, at 10:30, Jan Chochol wrote: > > Hi Martin, > > This is bug with combination of new zlib and old Erlang - https://github.com/erlang/otp/commit/e27119948fc6ab28bea81019720bddaac5b655a7 . > It is fixed in OTP-18.3.4.5 and newer. > "--enable-builtin-zlib" works, because it uses older bundled zlib. > > Jan > > On Mon, Nov 13, 2017 at 10:01 PM, Martin Sumner > wrote: > Rather foolishly I accepted the upgrade to High Sierra on my Macbook last night. This has led to an interesting problem with a binary_to_term line on my project. > > Following the upgrade, tests have started failing on this line: > > https://github.com/martinsumner/leveled/blob/master/src/leveled_sst.erl#L875 > > with error:badarg being thrown. > > However if I copy and paste the binary on which the badarg has been thrown out of the crash report, and into a erlang console - binary_to_term works just fine. > > See this issue for an example - https://github.com/martinsumner/leveled/issues/106 > > This failure is intermittent, and indeed wrapping the binary_to_term like this is enough to make the problem seemingly disappear: > > try binary_to_term(Bin) of > T -> > T > catch > error:badarg -> > binary_to_term(Bin) > end. > > Re-running the tests on another OSX device on a release previous to High Sierra shows no issues with/without the wrapper. > > The problem appears to consistently exist in OTP 17.5 and 18.3. However, running the tests in 19.3 and R16B02-basho10 doesn't show the problem. > > The terms in question have all been compressed using term_to_binary's zlib compression. > > If I build 18.3 on High Sierra with the "--enable-builtin-zlib" option, the problem does *not* re-occur. This switch appears to be a reliable delta between having and not having the problem, and I believe may have become the default in OTP 17. > > It appears to be specific to v17/v18 Erlang with compressed terms, on High Sierra where the OS zlib implementation is used, not the inbuilt OTP zlib. > > There appears to be internet chatter of other issues with Mac OSX High Sierra and changes to the zlib implementation - https://github.com/madler/zlib/issues/305 . > > So I think this doesn't look like it is an Erlang issue, but I suspect it may not be resolved in High Sierra any time soon, and may impact other Erlang users using zlib functionality. Does this it seem reasonable to pass this off as a High Sierra zlib bug, and use the --enable-builtin-zlib workaround? If this is the case I'm not sure why I don't get the issue on 19.3, any ideas as to why? > > Martin > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Tue Nov 14 09:50:59 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 14 Nov 2017 09:50:59 +0100 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: Message-ID: Most operators have a corresponding function in the 'erlang' module. For example, A + B can be written erlang:'+'(A,B). However, the cons operator [|] does not, as far as I know. /Richard 2017-11-14 8:56 GMT+01:00 Zhongzheng Liu : > Hi mail-list: > > Which built-in function work as fun(H, T) -> [H|T] end ? > > i.e how to fill the following expression without user defined function ? > > [H|T] = erlang:apply(???, ???, [H, T]). > > > Thanks > > Liu zhongzheng > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From meruiz77@REDACTED Tue Nov 14 09:48:17 2017 From: meruiz77@REDACTED (Miguel Emilio Ruiz) Date: Tue, 14 Nov 2017 09:48:17 +0100 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: Message-ID: <2434F3A4-0380-43EC-9B6A-6548EE84882B@gmail.com> Hi, I don't know if what you want is this, I'm newbie in Erlang: [H|T] = [H] ++ T. Miguel Ruiz > El 14 nov 2017, a las 8:56, Zhongzheng Liu escribi?: > > Hi mail-list: > > Which built-in function work as fun(H, T) -> [H|T] end ? > > i.e how to fill the following expression without user defined function ? > > [H|T] = erlang:apply(???, ???, [H, T]). > > > Thanks > > Liu zhongzheng > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From roger@REDACTED Tue Nov 14 10:55:48 2017 From: roger@REDACTED (Roger Lipscombe) Date: Tue, 14 Nov 2017 09:55:48 +0000 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: Message-ID: See this thread from earlier this year: http://erlang.org/pipermail/erlang-questions/2017-January/091560.html On 14 November 2017 at 08:50, Richard Carlsson wrote: > Most operators have a corresponding function in the 'erlang' module. For > example, A + B can be written erlang:'+'(A,B). However, the cons operator > [|] does not, as far as I know. > > > > > /Richard > > 2017-11-14 8:56 GMT+01:00 Zhongzheng Liu : >> >> Hi mail-list: >> >> Which built-in function work as fun(H, T) -> [H|T] end ? >> >> i.e how to fill the following expression without user defined function ? >> >> [H|T] = erlang:apply(???, ???, [H, T]). >> >> >> Thanks >> >> Liu zhongzheng >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From pierrefenoll@REDACTED Tue Nov 14 11:04:47 2017 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Tue, 14 Nov 2017 10:04:47 +0000 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: Message-ID: It would be nice to have a cons function, either in the erlang or lists module. One usage for it I have had is as the accumulator function of filelib:fold_dirs/5 On Tue 14 Nov 2017 at 10:55, Roger Lipscombe wrote: > See this thread from earlier this year: > http://erlang.org/pipermail/erlang-questions/2017-January/091560.html > > On 14 November 2017 at 08:50, Richard Carlsson > wrote: > > Most operators have a corresponding function in the 'erlang' module. For > > example, A + B can be written erlang:'+'(A,B). However, the cons operator > > [|] does not, as far as I know. > > > > > > > > > > /Richard > > > > 2017-11-14 8:56 GMT+01:00 Zhongzheng Liu : > >> > >> Hi mail-list: > >> > >> Which built-in function work as fun(H, T) -> [H|T] end ? > >> > >> i.e how to fill the following expression without user defined function ? > >> > >> [H|T] = erlang:apply(???, ???, [H, T]). > >> > >> > >> Thanks > >> > >> Liu zhongzheng > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Cheers, -- Pierre Fenoll -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Tue Nov 14 11:15:12 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Tue, 14 Nov 2017 11:15:12 +0100 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: Message-ID: Operation [H|T] and [H] ++ [T] probably too, is replaced by the compiler with a `put_list` instruction. If you want it to be callable for high order functions it must be located inside some fun, so give it a lambda: fun(H, T) -> [H|T] end, or create your own function, which will conveniently compile into: func_info some_name 2 (the arity) put_list X0 X1 X0 return or something like this. And it will become usable in HOFs now 2017-11-14 11:04 GMT+01:00 Pierre Fenoll : > It would be nice to have a cons function, either in the erlang or lists > module. > One usage for it I have had is as the accumulator function of > filelib:fold_dirs/5 > > On Tue 14 Nov 2017 at 10:55, Roger Lipscombe > wrote: > >> See this thread from earlier this year: >> http://erlang.org/pipermail/erlang-questions/2017-January/091560.html >> >> On 14 November 2017 at 08:50, Richard Carlsson >> wrote: >> > Most operators have a corresponding function in the 'erlang' module. For >> > example, A + B can be written erlang:'+'(A,B). However, the cons >> operator >> > [|] does not, as far as I know. >> > >> > >> > >> > >> > /Richard >> > >> > 2017-11-14 8:56 GMT+01:00 Zhongzheng Liu : >> >> >> >> Hi mail-list: >> >> >> >> Which built-in function work as fun(H, T) -> [H|T] end ? >> >> >> >> i.e how to fill the following expression without user defined function >> ? >> >> >> >> [H|T] = erlang:apply(???, ???, [H, T]). >> >> >> >> >> >> Thanks >> >> >> >> Liu zhongzheng >> >> _______________________________________________ >> >> erlang-questions mailing list >> >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> > >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- > > Cheers, > -- > Pierre Fenoll > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cons@REDACTED Tue Nov 14 11:15:14 2017 From: cons@REDACTED (=?utf-8?Q?Cons_T_=C3=85hs?=) Date: Tue, 14 Nov 2017 11:15:14 +0100 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: Message-ID: <69782863-5E42-4191-9933-2F04BA6BB5BA@tail-f.com> > On 14 Nov 2017, at 09:50, Richard Carlsson wrote: > > Most operators have a corresponding function in the 'erlang' module. For example, A + B can be written erlang:'+'(A,B). However, the cons operator [|] does not, as far as I know. I?ve always considered this a gross oversight for several reasons. One is that you have to write your own local fun when you want to pass it as higher order functional argument, another is that should one generate Erlang code directly one has to take care of this and generate special syntax. Then there is the personal obvious reason :-) Cons > > > > > /Richard > > 2017-11-14 8:56 GMT+01:00 Zhongzheng Liu >: > Hi mail-list: > > Which built-in function work as fun(H, T) -> [H|T] end ? > > i.e how to fill the following expression without user defined function ? > > [H|T] = erlang:apply(???, ???, [H, T]). > > > Thanks > > Liu zhongzheng > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Tue Nov 14 11:46:03 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 14 Nov 2017 11:46:03 +0100 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: <69782863-5E42-4191-9933-2F04BA6BB5BA@tail-f.com> References: <69782863-5E42-4191-9933-2F04BA6BB5BA@tail-f.com> Message-ID: I'll make sure it gets named erlang:'[_|_]'/2 in your honour. /Richard 2017-11-14 11:15 GMT+01:00 Cons T ?hs : > > On 14 Nov 2017, at 09:50, Richard Carlsson > wrote: > > Most operators have a corresponding function in the 'erlang' module. For > example, A + B can be written erlang:'+'(A,B). However, the cons operator > [|] does not, as far as I know. > > > I?ve always considered this a gross oversight for several reasons. One is > that you have to write your own local fun when you want to pass it as > higher order functional argument, another is that should one generate > Erlang code directly one has to take care of this and generate special > syntax. Then there is the personal obvious reason :-) > > Cons > > > > > > /Richard > > 2017-11-14 8:56 GMT+01:00 Zhongzheng Liu : > >> Hi mail-list: >> >> Which built-in function work as fun(H, T) -> [H|T] end ? >> >> i.e how to fill the following expression without user defined function ? >> >> [H|T] = erlang:apply(???, ???, [H, T]). >> >> >> Thanks >> >> Liu zhongzheng >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Nov 14 23:00:28 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 15 Nov 2017 11:00:28 +1300 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: Message-ID: <959fb10b-251d-1ff2-b185-def32aa56ea6@cs.otago.ac.nz> On 14/11/17 9:50 PM, Richard Carlsson wrote: > Most operators have a corresponding function in the 'erlang' module. For > example, A + B can be written erlang:'+'(A,B). However, the cons > operator [|] does not, as far as I know. I suggest rolling your own. cons(H, T) -> [H|T]. erlang:apply(?MODULE, cons, [H,T]) From facundo.olano@REDACTED Tue Nov 14 23:08:16 2017 From: facundo.olano@REDACTED (Facundo Olano) Date: Tue, 14 Nov 2017 19:08:16 -0300 Subject: [erlang-questions] [ANN] HolidayPing released Message-ID: Hi everyone! My company just released an open source web application implemented in Erlang. HolidayPing is a small web application that sends holiday reminders through different services like email and slack. It?s mainly aimed at consultants and freelancers that work with clients abroad. - HolidayPing application . - Article about the development process . - Source code at GitHub . Thanks, Facundo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Nov 14 23:12:45 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 15 Nov 2017 11:12:45 +1300 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: Message-ID: On reflection, the question needs to be asked: "why do you want to do this"? It's not wrong or stupid or anything, it just seems like a rather low-level thing to be passing around. I'd normally expect a function parameter to be a bit more, well, application-specific. What I'm getting at is that there might be a better way to do whatever it is you really want to do. From ok@REDACTED Wed Nov 15 05:32:05 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 15 Nov 2017 17:32:05 +1300 Subject: [erlang-questions] E-mail address change process? In-Reply-To: References: Message-ID: How do you change your e-mail address for this mailing list? I will be switching from this address to a gmail one soon. From raimo+erlang-questions@REDACTED Wed Nov 15 10:20:25 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 15 Nov 2017 10:20:25 +0100 Subject: [erlang-questions] E-mail address change process? In-Reply-To: References: Message-ID: <20171115092025.GA86416@erix.ericsson.se> On Wed, Nov 15, 2017 at 05:32:05PM +1300, Richard A. O'Keefe wrote: > How do you change your e-mail address for this mailing list? > I will be switching from this address to a gmail one soon. You subscribe the new and cancel the old. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From a@REDACTED Wed Nov 15 10:30:00 2017 From: a@REDACTED (Alexander Mihajlovic) Date: Wed, 15 Nov 2017 10:30:00 +0100 Subject: [erlang-questions] E-mail address change process? In-Reply-To: References: Message-ID: <1510738200.1722099.1173131336.44EC4ABF@webmail.messagingengine.com> Go to http://erlang.org/mailman/listinfo/erlang-questions At the bottom of that page, click the button "Unsubscribe or edit options" Log in (if you're like me you'll have to request a password reminder), and use the form to change your address. On Wed, Nov 15, 2017, at 05:32, Richard A. O'Keefe wrote: > How do you change your e-mail address for this mailing list? > I will be switching from this address to a gmail one soon. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From frank.muller.erl@REDACTED Wed Nov 15 13:31:01 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 15 Nov 2017 12:31:01 +0000 Subject: [erlang-questions] beam.smp high memory usage In-Reply-To: References: Message-ID: Hi guys, I would like to thank you for your feedbacks and pointers. After days of unsuccessful investigations, Jesper?s hint led me directly the culprit. All clients were sending small binaries to my app, but one. This one guy sent very large binaries for very long time (keep-alive). A temporary workaround was to simple limit the size of each binary sent and force closing TCP connection after N seconds. The memory usage went immediately back to ~160MB. Thanks again /Frank Fred's excellent "Erlang In Anger" book (https://www.erlang-in-anger.com/) > has a section on hunting down binary leaks (if that is what you are working > with here). At least this can give you some starting information about what > to look for and how to go about handling it afterwards. Identifying which > processes are holding the binary data by extension identifies the code > modules which are culprits. > > Since your normal operation uses 160 megabytes of memory and it suddenly > shoots to gigabytes, one of the parts is to identify what is being done to > the system when that happens. Perhaps you have an outside user which uses > your system in a way that wasn't thought about originally. This can lead to > you handling different data and in turn this can expose some of these > problems. > > On Mon, Nov 13, 2017 at 9:23 AM Frank Muller > wrote: > >> Hi Jesper, >> >> Thanks again for the feedback. >> >> But from where should I start (Erlang newbie here)? >> I?ve no idea ... is there a guideline to follow? >> >> I feel guessing without making any real progress for the last two days :-/ >> >> /Frank >> >> First quick stab: >>> >>> Most of the allocated space is binary() data. >>> >>> Typical scenario: something is keeping the binary() data alive, one way >>> or the other. Either because you need a binary:copy/1 in key locations due >>> to sub-binaries, or because your system can't get rid of the data since you >>> still have a pointer kept to it from some process. >>> >>> >>> On Sun, Nov 12, 2017 at 12:51 PM Frank Muller < >>> frank.muller.erl@REDACTED> wrote: >>> >>>> Hi guys, >>>> >>>> I?ve a fairly complex I/O bound Erlang application. >>>> >>>> Since yesterday, the beam.smp memory usage dramatically increased from >>>> ~160MB to something between [2.5GB, 3.1GB]. >>>> I?ve no idea how, nor why :-/ >>>> >>>> Attached is snapshot taken with recon_alloc:snapshot(). >>>> Please let me know if you want anything else. >>>> >>>> Hope someone can explain me what?s going on. >>>> >>>> Thank you. >>>> /Frank >>>> >>>> INFO: >>>> . Physical machine (not a VM) >>>> . Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] >>>> [async-threads:10] [hipe] [kernel-poll:false] >>>> . CentOS-7-x86_64 >>>> . Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 >>>> CEST 2017 x86_64 x86_64 x86_64 GNU/Linux >>>> . 8x Intel(R) Xeon(R) CPU E5504 @ 2.00GHz >>>> . 16GB of RAM >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gordeev.vladimir.v@REDACTED Wed Nov 15 15:33:49 2017 From: gordeev.vladimir.v@REDACTED (Vladimir Gordeev) Date: Wed, 15 Nov 2017 17:33:49 +0300 Subject: [erlang-questions] erlang:trace_pattern/3 is not expected to work on ports? Message-ID: Currently trying to match received messages for ports, do the following: erlang:trace_pattern('receive', [MyComplexMatchSpec], []), erlang:trace(ports, true, [ports, timestamp, 'receive', {tracer, self()}]), Got all messages received by port, filtering doesn't work. Okay, tried this one: erlang:trace_pattern('receive', false, []), erlang:trace(ports, true, [ports, timestamp, 'receive', {tracer, self()}]), And still got all messages, while shouldn't receive any. So, erlang:trace_pattern/3 is not supposed to work for ports? In official docs only processes are mentioned, but no explicit statement about ports. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Nov 15 21:04:24 2017 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 15 Nov 2017 15:04:24 -0500 (EST) Subject: [erlang-questions] How best to extract value from mnesia query Message-ID: <1510776264.84232835@apps.rackspace.com> Hello, The result of a mnesia read transaction looks like this: {atomic, Results} where result is either a populated list [value1, ... valueN] or, an empty list []. If the table is initialized as set, then the returned list will contain at most one value, e.g: [value]. If I want to use of the value in a function or to populate another record, I need to extract the value from the list. I've tried various ways to do this, but they all seem clumsy: E.g. [MyValue] hd[vlaue] confirm([]) -> not_found; confirm([Value]) -> Value; confirm(Value) -> Value. Does there happen to be a preferred/conventional/best-practices way to do this? Many thanks, LRP From jesper.louis.andersen@REDACTED Wed Nov 15 22:44:53 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 15 Nov 2017 21:44:53 +0000 Subject: [erlang-questions] How best to extract value from mnesia query In-Reply-To: <1510776264.84232835@apps.rackspace.com> References: <1510776264.84232835@apps.rackspace.com> Message-ID: I usually do something like: Txn = fun() -> ... end, case mnesia:transaction(Txn) of {atomic, []} -> not_found; {atomic, [V]} -> {atomic, L = [_|_]} -> end. Some times, if there are a bunch of cases, I tend to use a function such as your confirm/1 function, passing the result on and writing down the case split at the top level, but it depends a bit on wether I like that code flow or not in that particular case. In general I prefer matching and binding values over projection functions such as hd/1, because hd/1 doesn't work on all lists, only the non-empty ones. Another common trick is that if your Results are lists, then you can often avoid using a not_found atom, but handle the [] case naturally. Consider that [] = lists:map(F, []) which is true for a lot of functions working on lists. This suggests you can avoid having multiple data flows in this case, and use [] as a degenerate case which "skips" computations in a natural way. This can sometimes turn a code flow which case-splits its control flow all the time into a flow that just has one path. Flow with a single path tend to be far easier to handle in the long run in my experience. Also, if you really do expect there to be a value, then I use {atomic, [Value]} = mnesia:transaction(Txn), and crash the code if it is violated. Defensive code is often a mistake in Erlang because we can just use the standard crash semantics to recover. Many other languages doesn't fare so well here. On Wed, Nov 15, 2017 at 9:04 PM wrote: > Hello, > > The result of a mnesia read transaction looks like this: > > {atomic, Results} where result is either a populated list [value1, ... > valueN] or, an empty list []. > > If the table is initialized as set, then the returned list will contain at > most one value, e.g: > > [value]. > > If I want to use of the value in a function or to populate another record, > I need to extract the value from the list. > > I've tried various ways to do this, but they all seem clumsy: > > E.g. > > [MyValue] > > hd[vlaue] > > confirm([]) -> > not_found; > confirm([Value]) -> > Value; > confirm(Value) -> > Value. > > Does there happen to be a preferred/conventional/best-practices way to do > this? > > Many thanks, > > LRP > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Wed Nov 15 22:46:43 2017 From: rtrlists@REDACTED (Robert Raschke) Date: Wed, 15 Nov 2017 22:46:43 +0100 Subject: [erlang-questions] How best to extract value from mnesia query In-Reply-To: <1510776264.84232835@apps.rackspace.com> References: <1510776264.84232835@apps.rackspace.com> Message-ID: If I know that I shall only receive a single value result, I write {atomic, [Value]} = mnesia:transaction(...) This has the side effect of neatly terminating the process if we didn't get exactly one value back as a result. And that allows me to catch the error and decide what to do about it. If you want to write defensively, then something that discriminates between the results may be appropriate. case mnesia:transaction(...) of {aborted, Reason} -> ...; {atomic, []} -> ...; {atomic, [Value]} -> ...; {atomic, Values} -> ... end Cheers, Robby On 15 Nov 2017 21:04, wrote: Hello, The result of a mnesia read transaction looks like this: {atomic, Results} where result is either a populated list [value1, ... valueN] or, an empty list []. If the table is initialized as set, then the returned list will contain at most one value, e.g: [value]. If I want to use of the value in a function or to populate another record, I need to extract the value from the list. I've tried various ways to do this, but they all seem clumsy: E.g. [MyValue] hd[vlaue] confirm([]) -> not_found; confirm([Value]) -> Value; confirm(Value) -> Value. Does there happen to be a preferred/conventional/best-practices way to do this? Many thanks, LRP _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Thu Nov 16 00:31:58 2017 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 15 Nov 2017 18:31:58 -0500 (EST) Subject: [erlang-questions] How best to extract value from mnesia query In-Reply-To: References: <1510776264.84232835@apps.rackspace.com> Message-ID: <1510788718.95620556@apps.rackspace.com> Hi Jesper, Thank you! Lloyd -----Original Message----- From: "Jesper Louis Andersen" Sent: Wednesday, November 15, 2017 4:44pm To: lloyd@REDACTED Cc: "Erlang" Subject: Re: [erlang-questions] How best to extract value from mnesia query I usually do something like: Txn = fun() -> ... end, case mnesia:transaction(Txn) of {atomic, []} -> not_found; {atomic, [V]} -> {atomic, L = [_|_]} -> end. Some times, if there are a bunch of cases, I tend to use a function such as your confirm/1 function, passing the result on and writing down the case split at the top level, but it depends a bit on wether I like that code flow or not in that particular case. In general I prefer matching and binding values over projection functions such as hd/1, because hd/1 doesn't work on all lists, only the non-empty ones. Another common trick is that if your Results are lists, then you can often avoid using a not_found atom, but handle the [] case naturally. Consider that [] = lists:map(F, []) which is true for a lot of functions working on lists. This suggests you can avoid having multiple data flows in this case, and use [] as a degenerate case which "skips" computations in a natural way. This can sometimes turn a code flow which case-splits its control flow all the time into a flow that just has one path. Flow with a single path tend to be far easier to handle in the long run in my experience. Also, if you really do expect there to be a value, then I use {atomic, [Value]} = mnesia:transaction(Txn), and crash the code if it is violated. Defensive code is often a mistake in Erlang because we can just use the standard crash semantics to recover. Many other languages doesn't fare so well here. On Wed, Nov 15, 2017 at 9:04 PM wrote: > Hello, > > The result of a mnesia read transaction looks like this: > > {atomic, Results} where result is either a populated list [value1, ... > valueN] or, an empty list []. > > If the table is initialized as set, then the returned list will contain at > most one value, e.g: > > [value]. > > If I want to use of the value in a function or to populate another record, > I need to extract the value from the list. > > I've tried various ways to do this, but they all seem clumsy: > > E.g. > > [MyValue] > > hd[vlaue] > > confirm([]) -> > not_found; > confirm([Value]) -> > Value; > confirm(Value) -> > Value. > > Does there happen to be a preferred/conventional/best-practices way to do > this? > > Many thanks, > > LRP > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From lloyd@REDACTED Thu Nov 16 00:34:18 2017 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 15 Nov 2017 18:34:18 -0500 (EST) Subject: [erlang-questions] How best to extract value from mnesia query In-Reply-To: References: <1510776264.84232835@apps.rackspace.com> Message-ID: <1510788858.06264092@apps.rackspace.com> Hi Robby, Much appreciate your helpful response. Lloyd -----Original Message----- From: "Robert Raschke" Sent: Wednesday, November 15, 2017 4:46pm To: lloyd@REDACTED Cc: "Erlang Questions" Subject: Re: [erlang-questions] How best to extract value from mnesia query If I know that I shall only receive a single value result, I write {atomic, [Value]} = mnesia:transaction(...) This has the side effect of neatly terminating the process if we didn't get exactly one value back as a result. And that allows me to catch the error and decide what to do about it. If you want to write defensively, then something that discriminates between the results may be appropriate. case mnesia:transaction(...) of {aborted, Reason} -> ...; {atomic, []} -> ...; {atomic, [Value]} -> ...; {atomic, Values} -> ... end Cheers, Robby On 15 Nov 2017 21:04, wrote: Hello, The result of a mnesia read transaction looks like this: {atomic, Results} where result is either a populated list [value1, ... valueN] or, an empty list []. If the table is initialized as set, then the returned list will contain at most one value, e.g: [value]. If I want to use of the value in a function or to populate another record, I need to extract the value from the list. I've tried various ways to do this, but they all seem clumsy: E.g. [MyValue] hd[vlaue] confirm([]) -> not_found; confirm([Value]) -> Value; confirm(Value) -> Value. Does there happen to be a preferred/conventional/best-practices way to do this? Many thanks, LRP _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From liuzhongzheng2012@REDACTED Thu Nov 16 03:19:15 2017 From: liuzhongzheng2012@REDACTED (Zhongzheng Liu) Date: Thu, 16 Nov 2017 10:19:15 +0800 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: <69782863-5E42-4191-9933-2F04BA6BB5BA@tail-f.com> References: <69782863-5E42-4191-9933-2F04BA6BB5BA@tail-f.com> Message-ID: As far as I know, there is no constructor function for list or tuple. We have maps:new(), why not lists:new() or tuple:new() ? Liu Zhongzheng 2017-11-14 18:15 GMT+08:00 Cons T ?hs : > > On 14 Nov 2017, at 09:50, Richard Carlsson > wrote: > > Most operators have a corresponding function in the 'erlang' module. For > example, A + B can be written erlang:'+'(A,B). However, the cons operator > [|] does not, as far as I know. > > > I?ve always considered this a gross oversight for several reasons. One is > that you have to write your own local fun when you want to pass it as higher > order functional argument, another is that should one generate Erlang code > directly one has to take care of this and generate special syntax. Then > there is the personal obvious reason :-) > > Cons > > > > > > /Richard > > 2017-11-14 8:56 GMT+01:00 Zhongzheng Liu : >> >> Hi mail-list: >> >> Which built-in function work as fun(H, T) -> [H|T] end ? >> >> i.e how to fill the following expression without user defined function ? >> >> [H|T] = erlang:apply(???, ???, [H, T]). >> >> >> Thanks >> >> Liu zhongzheng >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From atulindore2@REDACTED Thu Nov 16 08:27:57 2017 From: atulindore2@REDACTED (Atul Goyal) Date: Thu, 16 Nov 2017 12:57:57 +0530 Subject: [erlang-questions] sleep for a few microsconds Message-ID: hi I was wondering if there is a way to sleep for few u'sec in erlang . We are working on Linux. And we have tried following solutions : 1. Used linux 'nanosleep' from a 'C' NIF :: Gives 50-90% error between 1-500 microsec ... and even at 1500 microsec it gives 10-20% error . I tried giving high priority to Beam process .. same results 2.Busywait in Erlang function :: Isnt practical to waste CPU time when serving a large number of users ...Also it is only good for about 90% of the time but rest of the time gives 50-200% error ... thanks a lot Atul -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Nov 16 09:25:51 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 16 Nov 2017 11:25:51 +0300 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: References: Message-ID: Afraid that it is not possible. Perhaps the only way is to have a thread that will execute your commands with a nanosleep precision. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vances@REDACTED Thu Nov 16 09:36:45 2017 From: vances@REDACTED (Vance Shipley) Date: Thu, 16 Nov 2017 14:06:45 +0530 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: References: Message-ID: On Thu, Nov 16, 2017 at 12:57 PM, Atul Goyal wrote: > I was wondering if there is a way to sleep for few u'sec in erlang . Do the opposite? 1> F = fun(F, T1, T2) when T1 < T2 -> 1> F(F, erlang:system_time(milli_seconds), T2); 1> (_, _, _) -> 1> ok 1> end, 1> T = erlang:system_time(milli_seconds), 1> F(F, T, T + 3). ok -- -Vance From atulindore2@REDACTED Thu Nov 16 09:38:04 2017 From: atulindore2@REDACTED (Atul Goyal) Date: Thu, 16 Nov 2017 14:08:04 +0530 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: References: Message-ID: do u mean using new OS thread which might be spawned from say a linked in driver ? Thanks On Thu, Nov 16, 2017 at 1:55 PM, Max Lapshin wrote: > Afraid that it is not possible. > > Perhaps the only way is to have a thread that will execute your commands > with a nanosleep precision. > -- atul -------------- next part -------------- An HTML attachment was scrubbed... URL: From atulindore2@REDACTED Thu Nov 16 09:53:27 2017 From: atulindore2@REDACTED (Atul Goyal) Date: Thu, 16 Nov 2017 14:23:27 +0530 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: References: Message-ID: hi Vance Thanks but already tried that :) On Thu, Nov 16, 2017 at 2:06 PM, Vance Shipley wrote: > On Thu, Nov 16, 2017 at 12:57 PM, Atul Goyal > wrote: > > I was wondering if there is a way to sleep for few u'sec in erlang . > > Do the opposite? > > 1> F = fun(F, T1, T2) when T1 < T2 -> > 1> F(F, erlang:system_time(milli_seconds), T2); > 1> (_, _, _) -> > 1> ok > 1> end, > 1> T = erlang:system_time(milli_seconds), > 1> F(F, T, T + 3). > ok > > > -- > -Vance > -- atul -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Thu Nov 16 10:27:31 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 16 Nov 2017 10:27:31 +0100 Subject: [erlang-questions] Which built-in function makes [H|T] ? In-Reply-To: References: <69782863-5E42-4191-9933-2F04BA6BB5BA@tail-f.com> Message-ID: If you want to think in terms of constructors, where you first get an initial value and then "populate" it, then the constructor for lists is written [] (the empty list), and for a tuple you can call erlang:make_tuple(Arity, InitialValue) or use erlang:list_to_tuple(Elements). The function maps:new() simply returns #{}, so why it exists at all is a bit of a mystery. Maybe there were plans to add a maps:new(Options) which could allow you to set special flags, like in array:new(Options). /Richard 2017-11-16 3:19 GMT+01:00 Zhongzheng Liu : > As far as I know, there is no constructor function for list or tuple. > > We have maps:new(), why not lists:new() or tuple:new() ? > > Liu Zhongzheng > > > 2017-11-14 18:15 GMT+08:00 Cons T ?hs : > > > > On 14 Nov 2017, at 09:50, Richard Carlsson > > wrote: > > > > Most operators have a corresponding function in the 'erlang' module. For > > example, A + B can be written erlang:'+'(A,B). However, the cons operator > > [|] does not, as far as I know. > > > > > > I?ve always considered this a gross oversight for several reasons. One > is > > that you have to write your own local fun when you want to pass it as > higher > > order functional argument, another is that should one generate Erlang > code > > directly one has to take care of this and generate special syntax. Then > > there is the personal obvious reason :-) > > > > Cons > > > > > > > > > > > > /Richard > > > > 2017-11-14 8:56 GMT+01:00 Zhongzheng Liu : > >> > >> Hi mail-list: > >> > >> Which built-in function work as fun(H, T) -> [H|T] end ? > >> > >> i.e how to fill the following expression without user defined function ? > >> > >> [H|T] = erlang:apply(???, ???, [H, T]). > >> > >> > >> Thanks > >> > >> Liu zhongzheng > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Thu Nov 16 12:29:30 2017 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Thu, 16 Nov 2017 12:29:30 +0100 Subject: [erlang-questions] Patch package OTP 18.3.4.6 released Message-ID: <2ec5ffaa-ce9c-3ad8-9940-e62b27044957@ericsson.com> Patch Package: OTP 18.3.4.6 Git Tag: OTP-18.3.4.6 Date: 2017-11-16 Trouble Report Id: OTP-14491, OTP-14514, OTP-14522, OTP-14548, OTP-14590, OTP-14763, OTP-14765 Seq num: ERIERL-48, ERIERL-74, ERL-468, ERL-474 System: OTP Release: 18 Application: compiler-6.0.3.1, eldap-1.2.1.1, erts-7.3.1.4, ssh-4.2.2.4 Predecessor: OTP 18.3.4.5 Check out the git tag OTP-18.3.4.6, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- compiler-6.0.3.1 ------------------------------------------------ --------------------------------------------------------------------- The compiler-6.0.3.1 application can be applied independently of other applications on a full OTP 18 installation. --- Fixed Bugs and Malfunctions --- OTP-14522 Application(s): compiler Related Id(s): ERIERL-48 Fail labels on guard BIFs weren't taken into account during an optimization pass, and a bug in the validation pass sometimes prevented this from being noticed when a fault occurred. Full runtime dependencies of compiler-6.0.3.1: crypto-3.6, erts-7.0, hipe-3.12, kernel-4.0, stdlib-2.5 --------------------------------------------------------------------- --- eldap-1.2.1.1 --------------------------------------------------- --------------------------------------------------------------------- The eldap-1.2.1.1 application can be applied independently of other applications on a full OTP 18 installation. --- Improvements and New Features --- OTP-14765 Application(s): eldap, jinterface Misc building environment updates Full runtime dependencies of eldap-1.2.1.1: asn1-3.0, erts-6.0, kernel-3.0, ssl-5.3.4, stdlib-2.0 --------------------------------------------------------------------- --- erts-7.3.1.4 ---------------------------------------------------- --------------------------------------------------------------------- The erts-7.3.1.4 application can be applied independently of other applications on a full OTP 18 installation. --- Fixed Bugs and Malfunctions --- OTP-14491 Application(s): erts Fix performance bug in pre-allocators that could cause them to permanently fall back on normal more expensive memory allocation. Pre-allocators are used for quick allocation of short lived meta data used by messages and other scheduled tasks. Bug exists since OTP_R15B02. OTP-14514 Application(s): erts Fixed bug in operator bxor causing erroneuos result when one operand is a big *negative* integer with the lowest N*W bits as zero and the other operand not larger than N*W bits. N is an integer of 1 or larger and W is 32 or 64 depending on word size. OTP-14548 Application(s): erts Related Id(s): ERL-468, OTP-11997 A timer internal bit-field used for storing scheduler id was too small. As a result, VM internal timer data structures could become inconsistent when using 1024 schedulers on the system. Note that systems with less than 1024 schedulers are not effected by this bug. This bug was introduced in ERTS version 7.0 (OTP 18.0). OTP-14590 Application(s): erts Related Id(s): ERL-474 Fixed bug in binary_to_term and binary_to_atom that could cause VM crash. Typically happens when the last character of an UTF8 string is in the range 128 to 255, but truncated to only one byte. Bug exists in binary_to_term since ERTS version 5.10.2 (OTP_R16B01) and binary_to_atom since ERTS version 9.0 (OTP-20.0). Full runtime dependencies of erts-7.3.1.4: kernel-4.0, sasl-2.4, stdlib-2.5 --------------------------------------------------------------------- --- ssh-4.2.2.4 ----------------------------------------------------- --------------------------------------------------------------------- Note! The ssh-4.2.2.4 application can *not* be applied independently of other applications on an arbitrary OTP 18 installation. On a full OTP 18 installation, also the following runtime dependency has to be satisfied: -- crypto-3.6.3.1 (first satisfied in OTP 18.3.4.5) --- Fixed Bugs and Malfunctions --- OTP-14763 Application(s): ssh Related Id(s): ERIERL-74 Trailing white space was removed at end of the hello-string. This caused interoperability problems with some other ssh-implementations (e.g OpenSSH 7.3p1 on Solaris 11) Full runtime dependencies of ssh-4.2.2.4: crypto-3.6.3.1, erts-6.0, kernel-3.0, public_key-0.22, stdlib-2.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- Mailto: erlang-questions@REDACTED Subject: Patch package OTP 18.3.4.6 released Body: The OTP 18.3.4.6_open_src.README text * * Verify that the git tag OTP-18.3.4.6 has been pushed. * NOTE: YOU WILL HAVE TO EXPORT THE WEBPAGES: exit the otp user shell cd /usr/local/otp/releases/PATCHES/ && gmake cd ~erlang/www/erlang.se/scripts && ./update_erlang.se PATCHES (you need passwd for user 'otpwww') -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4101 bytes Desc: S/MIME Cryptographic Signature URL: From max.lapshin@REDACTED Thu Nov 16 17:22:29 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 16 Nov 2017 19:22:29 +0300 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: References: Message-ID: Yes. I mean separate thread, spawned from driver or nif that will consume your jobs with timings and do required sleep. -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact@REDACTED Thu Nov 16 19:30:57 2017 From: contact@REDACTED (Zachary N. Dean) Date: Thu, 16 Nov 2017 19:30:57 +0100 Subject: [erlang-questions] xqerl (pronounced squirrel) Message-ID: <002201d35f09$0d0cbf80$27263e80$@zadean.com> Hello world!! ?? I?m currently working on a project that needs some community input (https://github.com/zadean/xqerl). (? see what I did there with the "xq"erl)? It is an XQuery processor implemented in Erlang. That is? XQuery compiled to BEAM. I know, I know? a query language in Erlang??? XML??? Why?!!! Well, XQuery 3.1 is a functional language with all/most the bells-and-whistles (higher-order functions, maps/arrays, etc.), so I figured it fit, and I?d give it a go. It?s rough around the edges (and most-likely in the middle) - I know that - therefore, it has a lot of room to grow and improve. I?ve been doing everything myself, and have honestly never worked in an Erlang shop, so any advice or tips from the experts is more than welcome! Thanks in advance, Zack From essen@REDACTED Thu Nov 16 20:18:58 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 16 Nov 2017 20:18:58 +0100 Subject: [erlang-questions] Patch package OTP 18.3.4.6 released In-Reply-To: <2ec5ffaa-ce9c-3ad8-9940-e62b27044957@ericsson.com> References: <2ec5ffaa-ce9c-3ad8-9940-e62b27044957@ericsson.com> Message-ID: <686e88b9-9768-904d-0def-82ef6e09ea21@ninenines.eu> Hello, On 11/16/2017 12:29 PM, Hans Nilsson R wrote: > Patch Package: OTP 18.3.4.6 > Git Tag: OTP-18.3.4.6 I thought these tags were not supposed to be announced (that 18.3.4 would, but not 18.3.4.6). Did I miss something? Asking because I'm tracking patch releases in the ci.erlang.mk plugin[1] but if you're going to announce these the same way as patch releases then I probably should track those instead. [1] https://git.ninenines.eu/ci.erlang.mk.git/tree/early-plugins.mk > --------------------------------------------------------------------- > --------------------------------------------------------------------- > --------------------------------------------------------------------- > > > > Mailto: erlang-questions@REDACTED > > Subject: Patch package OTP 18.3.4.6 released > > Body: The OTP 18.3.4.6_open_src.README text > > > > * > * Verify that the git tag OTP-18.3.4.6 has been pushed. > * > > > NOTE: YOU WILL HAVE TO EXPORT THE WEBPAGES: > exit the otp user shell > cd /usr/local/otp/releases/PATCHES/ && gmake > cd ~erlang/www/erlang.se/scripts && ./update_erlang.se PATCHES > (you need passwd for user 'otpwww') What's the password for 'otpwww'? ;-) Cheers, -- Lo?c Hoguin https://ninenines.eu From sergej.jurecko@REDACTED Thu Nov 16 20:44:45 2017 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Thu, 16 Nov 2017 20:44:45 +0100 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: References: Message-ID: <7E5AF330-D551-4024-9B66-1032B1971E5D@gmail.com> Have you tried nanosleep accuracy in a simple C program by itself? Especially on a non-idle machine. Also I hope you?re not running or planning on running anything like that in a VM. Regards, Sergej > On 16 Nov 2017, at 08:27, Atul Goyal wrote: > > hi > > I was wondering if there is a way to sleep for few u'sec in erlang . We are working on Linux. And we have tried following solutions : > > 1. Used linux 'nanosleep' from a 'C' NIF :: > Gives 50-90% error between 1-500 microsec ... and even at 1500 microsec it gives 10-20% error . I tried giving high priority to Beam process .. same results > > 2.Busywait in Erlang function :: > Isnt practical to waste CPU time when serving a large number of users ...Also it is only good for about 90% of the time but rest of the time gives 50-200% error ... > > thanks a lot > Atul > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From atulindore2@REDACTED Thu Nov 16 21:27:13 2017 From: atulindore2@REDACTED (Atul Goyal) Date: Fri, 17 Nov 2017 01:57:13 +0530 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: <7E5AF330-D551-4024-9B66-1032B1971E5D@gmail.com> References: <7E5AF330-D551-4024-9B66-1032B1971E5D@gmail.com> Message-ID: Ya ... i checked nanosleep accuracy in pure c first .. with rt priority too ... it was quite accurate.. And i am using a physical machine . Thanks On 17-Nov-2017 01:14, "Sergej Jure?ko" wrote: > Have you tried nanosleep accuracy in a simple C program by itself? > Especially on a non-idle machine. Also I hope you?re not running or > planning on running anything like that in a VM. > > Regards, > Sergej > > > On 16 Nov 2017, at 08:27, Atul Goyal wrote: > > > > hi > > > > I was wondering if there is a way to sleep for few u'sec in erlang . We > are working on Linux. And we have tried following solutions : > > > > 1. Used linux 'nanosleep' from a 'C' NIF :: > > Gives 50-90% error between 1-500 microsec ... and even at 1500 microsec > it gives 10-20% error . I tried giving high priority to Beam process .. > same results > > > > 2.Busywait in Erlang function :: > > Isnt practical to waste CPU time when serving a large number of users > ...Also it is only good for about 90% of the time but rest of the time > gives 50-200% error ... > > > > thanks a lot > > Atul > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Nov 17 09:32:01 2017 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 17 Nov 2017 09:32:01 +0100 Subject: [erlang-questions] Patch package OTP 18.3.4.6 released In-Reply-To: <686e88b9-9768-904d-0def-82ef6e09ea21@ninenines.eu> References: <2ec5ffaa-ce9c-3ad8-9940-e62b27044957@ericsson.com> <686e88b9-9768-904d-0def-82ef6e09ea21@ninenines.eu> Message-ID: Hello, On Thu, Nov 16, 2017 at 8:18 PM, Lo?c Hoguin wrote: > Hello, > > On 11/16/2017 12:29 PM, Hans Nilsson R wrote: > >> Patch Package: OTP 18.3.4.6 >> Git Tag: OTP-18.3.4.6 >> > > I thought these tags were not supposed to be announced (that 18.3.4 would, > but not 18.3.4.6). Did I miss something? > When the next major release is released we branch the patch version in order to show that what is being patched in the previous version is not implicitly part of the next version. So in this case as long as OTP-19.0 was not released we patched OTP-18.3 by adding a third patch numeral and all patches were merged to be part of OTP-19.0. But when OTP-19.0 was released, all patches released to OTP-18.3 are not implicitly part of OTP-19.0 so we branch the version by adding a fourth numeral. So for all practical purposes 18.3.4.6 can be seen as 18.3.10, but because we branched the versioning it is possible to deduce which patches on OTP-18.3 that are part of 19.0 and which will be part of later versions of OTP-19.x. This is a bit confusing at first (at least it was for me...), but it embeds more information into the version numbers which can be useful. There is a description of the versioning scheme here: http://erlang.org/doc/system_ principles/versions.html And then to answer your question, yes we do announce these patches. We also sometimes do patches on not the tip of a previous major release, i.e. 18.2.3, this would then we 18.2.3.1 or some such. These patches we most likely will not announce. > > Asking because I'm tracking patch releases in the ci.erlang.mk plugin[1] > but if you're going to announce these the same way as patch releases then I > probably should track those instead. > You should definitely track the 18.3.4.x patches, but not the 18.2.x.y patches. The same goes for 19.3.6.x patches, and whatever 20.x.y.z we will end up at. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Nov 17 19:09:25 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Fri, 17 Nov 2017 19:09:25 +0100 Subject: [erlang-questions] Patch package OTP 18.3.4.6 released In-Reply-To: References: <2ec5ffaa-ce9c-3ad8-9940-e62b27044957@ericsson.com> <686e88b9-9768-904d-0def-82ef6e09ea21@ninenines.eu> Message-ID: On 11/17/2017 09:32 AM, Lukas Larsson wrote:> Asking because I'm tracking patch releases in the ci.erlang.mk > plugin[1] but if you're going to announce > these the same way as patch releases then I probably should track > those instead. > > > You should definitely track the 18.3.4.x patches, but not the 18.2.x.y > patches. The same goes for 19.3.6.x patches, and whatever 20.x.y.z we > will end up at. Will do! Thanks. -- Lo?c Hoguin https://ninenines.eu From roger@REDACTED Fri Nov 17 20:20:59 2017 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 17 Nov 2017 19:20:59 +0000 Subject: [erlang-questions] Paths in Common Test app.config Message-ID: It appears that Common Test runs my suites from an arbitrary current directory (somewhere under "logs"). This means that I cannot reliably use relative paths in my 'app.config' file. For example, I can't easily configure the paths to my TLS certificate files without monkeying around with the current working directory before calling application:ensure_started(my_app). Currently, I've got some code that walks back up the directory tree -- with file:set_cwd("..") -- until it finds a marker file (currently "rebar.config"), and then I specify the paths relative to this. This seems ... brittle. What do others do to deal with this? From eric.pailleau@REDACTED Fri Nov 17 20:50:57 2017 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Fri, 17 Nov 2017 20:50:57 +0100 Subject: [erlang-questions] Paths in Common Test app.config In-Reply-To: Message-ID: <276ce064-838f-45cc-8e68-5967ad423cf9@email.android.com> An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Fri Nov 17 20:58:31 2017 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Fri, 17 Nov 2017 20:58:31 +0100 Subject: [erlang-questions] Paths in Common Test app.config In-Reply-To: <276ce064-838f-45cc-8e68-5967ad423cf9@email.android.com> Message-ID: An HTML attachment was scrubbed... URL: From roger@REDACTED Fri Nov 17 21:06:37 2017 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 17 Nov 2017 20:06:37 +0000 Subject: [erlang-questions] Paths in Common Test app.config In-Reply-To: <276ce064-838f-45cc-8e68-5967ad423cf9@email.android.com> References: <276ce064-838f-45cc-8e68-5967ad423cf9@email.android.com> Message-ID: That doesn't seem to be what I want. Let me clarify... I run my tests with "rebar3 ct", and I have an app.config file containing the following (for example): [{my_app, [{cert_path, "relative/path/to/cert.crt"}]}]. It can't be an absolute path, because the project might be checked out in different locations on different machines. In my suite, I run my application as follows: application:ensure_started(my_app) I have, in my application: CertPath = application:get_env(my_app, cert_path) But, when "rebar3 ct" runs my test, the current directory is set to somewhere like "/home/roger/Projects/my_app/_build/test/logs/ct_run.nonode@REDACTED/", which means that these relative paths are wrong, and my application doesn't start correctly. When using rebar3, or CT itself, how do I control the initial working directory for the suite, so that I can put my cert files in a sensible location, and refer to them with relative paths? I'm considering using file:set_cwd(?config(data_dir, Config) ++ "/..") in each suite's init_per_testcase, but that feels awkward. To reiterate: I'm not trying to configure the test. I'm trying to configure the application under test. On 17 November 2017 at 19:50, ?ric Pailleau wrote: > Hi, > > Use config to set external things available > > http://erlang.org/doc/apps/common_test/config_file_chapter.html#id84119 > > Regards > > From eric.pailleau@REDACTED Fri Nov 17 22:51:11 2017 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Fri, 17 Nov 2017 22:51:11 +0100 Subject: [erlang-questions] Paths in Common Test app.config In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Sat Nov 18 00:21:30 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 17 Nov 2017 18:21:30 -0500 Subject: [erlang-questions] Paths in Common Test app.config In-Reply-To: References: Message-ID: <20171117232129.GA2024@ferdmbp.local> On 11/17, Roger Lipscombe wrote: >It appears that Common Test runs my suites from an arbitrary current >directory (somewhere under "logs"). > Yes, this allows any writes and reads taking place to be effectively in an isolated context for the test. This is generally a good thing since you can get per test run disk output to go look at how things are doing, what was created on disk and what was modified. >This means that I cannot reliably use relative paths in my >'app.config' file. For example, I can't easily configure the paths to >my TLS certificate files without monkeying around with the current >working directory before calling application:ensure_started(my_app). > >Currently, I've got some code that walks back up the directory tree -- >with file:set_cwd("..") -- until it finds a marker file (currently >"rebar.config"), and then I specify the paths relative to this. > >This seems ... brittle. > >What do others do to deal with this? It *is* brittle. I would try to avoid working with current working directory settings at pretty much all costs. The safe thing to do would be to give the certificates and files required in the data_dir for Common Test. This is a directory in test/ with the name your_SUITE_data/. In init_per_suite, you can then set the application env value: init_per_suite(Config) -> application:load(your_app), Cert = filename:join([?config(data_dir, Config), "some_cert.crt"]), applicatoin:set_env(your_app, certfile, Cert), Config. The test can then proceed as usual. If you really want to have the ability to 'carry' a file at all times with an app, that tends to be done through the priv/ directory of that application. From eric.pailleau@REDACTED Sat Nov 18 12:03:18 2017 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Sat, 18 Nov 2017 12:03:18 +0100 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: References: <7E5AF330-D551-4024-9B66-1032B1971E5D@gmail.com> Message-ID: <6fe640a6-7817-5cdb-e823-e5a1f5f2f947@wanadoo.fr> Hi, Erlang is soft real time, not hard realtime. I think it is hard to achieve an "exact" nanosleep in particular on multiCPU . Did you tried using a custom recursing function where argument is (somewhat) linear to execution time ? regards Le 16/11/2017 ? 21:27, Atul Goyal a ?crit?: > Ya ... i checked nanosleep accuracy in pure c first .. with rt priority > too ... it was quite accurate.. > And i am using a physical machine . > Thanks > > > On 17-Nov-2017 01:14, "Sergej Jure?ko" > wrote: > > Have you tried nanosleep accuracy in a simple C program by itself? > Especially on a non-idle machine. Also I hope you?re not running or > planning on running anything like that in a VM. > > Regards, > Sergej > > > On 16 Nov 2017, at 08:27, Atul Goyal > wrote: > > > > hi > > > > I was wondering if there is a way to sleep for few u'sec in > erlang . We are working on? Linux. And we have tried following > solutions : > > > > 1. Used linux 'nanosleep' from a 'C' NIF :: > > Gives 50-90% error between 1-500 microsec ... and even at 1500 > microsec it gives 10-20% error . I tried giving high priority to > Beam process .. same results > > > > 2.Busywait in Erlang function? :: > > Isnt practical to waste CPU time when serving a large number of > users ...Also it is only good for about 90% of the time but rest of > the time gives 50-200% error ... > > > > thanks a lot > > Atul > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From roger@REDACTED Sat Nov 18 14:17:02 2017 From: roger@REDACTED (Roger Lipscombe) Date: Sat, 18 Nov 2017 13:17:02 +0000 Subject: [erlang-questions] Paths in Common Test app.config In-Reply-To: <20171117232129.GA2024@ferdmbp.local> References: <20171117232129.GA2024@ferdmbp.local> Message-ID: On 17 November 2017 at 23:21, Fred Hebert wrote: > The safe thing to do would be to give the certificates and files required in > the data_dir for Common Test. This is a directory in test/ with the name > your_SUITE_data/. In init_per_suite, you can then set the application env > value: My problem is that I have several configuration settings that require filenames, at various levels of nesting. I also kinda wanted to share the cert files between all of the suites for the application. > init_per_suite(Config) -> > application:load(your_app), > Cert = filename:join([?config(data_dir, Config), "some_cert.crt"]), > applicatoin:set_env(your_app, certfile, Cert), > Config. It occurs that it might be possible to do something in a CT hook: it could recursively look for string values and do $VAR replacements on them from environment variables, or from that CT hook's options... > The test can then proceed as usual. If you really want to have the ability > to 'carry' a file at all times with an app, that tends to be done through > the priv/ directory of that application. Sure, but these are explicitly test artifacts; I'm not going to be keeping the production certificates and keys in the repository (*those* get installed by, e.g., chef during machine bring-up). From eric.pailleau@REDACTED Sat Nov 18 14:42:42 2017 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Sat, 18 Nov 2017 14:42:42 +0100 Subject: [erlang-questions] Paths in Common Test app.config In-Reply-To: Message-ID: <706211b8-7c40-419e-94aa-59ceeed3b12b@email.android.com> An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sat Nov 18 15:41:18 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 18 Nov 2017 14:41:18 +0000 Subject: [erlang-questions] sleep for a few microsconds In-Reply-To: References: Message-ID: On Thu, Nov 16, 2017 at 8:28 AM Atul Goyal wrote: > hi > > I was wondering if there is a way to sleep for few u'sec in erlang . We > are working on Linux. And we have tried following solutions : > > If you require microsecond precision sleeps, then chances are you won't be happy with either Erlang (or for that matter, Go). The problem is that in a highly concurrent system, with soft-realtime constraints, you can't be guaranteed a wakeup within a window every time. If you have 100000 processes on the run-queue, you will have to wait, one way or the other. Mind you, a hard realtime system wouldn't even allow you to start 100000 processes. It would start rejecting your spawns as soon as it can't make the guarantee. What Erlang systems *do* tend to do well in this regard is that the performance degrades linearly as the system becomes overloaded. That is, it is considerably harder to monopolize the CPU resources in an Erlang system compared to other systems (Haskell and Go fares well here too, but they require more scheduling points at the moment to be injected by the programmer). This is also the reason the precision is in milliseconds: it is far more likely that the Erlang system can meet the ms-window than the us or ns-windows. However, even that is no guarantee: a highly loaded system will fire timers late. The best way to cope with the lack of precision is to have a low-level precise system which you periodically feed from the Erlang world. Much like how you wake up and feed the small soundcard buffer once in a while to keep on playing. This is also how you can handle a mobile phone cell system: handle the low-level latency stuff elsewhere and use the Erlang system for orchestration. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilya.khaprov@REDACTED Sat Nov 18 20:20:52 2017 From: ilya.khaprov@REDACTED (Ilya Khaprov) Date: Sat, 18 Nov 2017 19:20:52 +0000 Subject: [erlang-questions] Removing eterm from NifEnv. Message-ID: Hello, I copy (enif_make_copy) an eterm to private NifEnv inside Nif. Later I want to remove that eterm from the NifEnv (actually I want to free memory, but I guess this means removing from NifEnv?s heap). How to do this? I know about enif_free_env, but this function clear entire env. Thank you, Ilya -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Sun Nov 19 23:51:41 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 20 Nov 2017 11:51:41 +1300 Subject: [erlang-questions] Paths in Common Test app.config In-Reply-To: References: <20171117232129.GA2024@ferdmbp.local> Message-ID: <76e2e46b-2238-cada-c50e-3dc73720cec0@cs.otago.ac.nz> On 19/11/17 2:17 AM, Roger Lipscombe wrote: > My problem is that I have several configuration settings that require > filenames, at various levels of nesting. I also kinda wanted to share > the cert files between all of the suites for the application. Can you put the cert files in a directory of their own and create symbolic links / aliases / whatever to them in the suites' data_dirs? From lukas@REDACTED Mon Nov 20 08:05:15 2017 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 20 Nov 2017 08:05:15 +0100 Subject: [erlang-questions] Removing eterm from NifEnv. In-Reply-To: References: Message-ID: Hello, On Sat, Nov 18, 2017 at 8:20 PM, Ilya Khaprov wrote: > I copy (enif_make_copy) an eterm to private NifEnv inside Nif. > > Later I want to remove that eterm from the NifEnv (actually I want to free > memory, but I guess this means removing from NifEnv?s heap). > > How to do this? I know about enif_free_env, but this function clear entire > env. > It is not possible to free an individual term from a nif environment. If you want to have that kind of granularity, you have to create one environment per term that you want to individually collect. If the allocation cost of a new env is too large for your application, you could pool env's and use enif_clear_env to re-cycle env's into the pool. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Mon Nov 20 12:21:45 2017 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Mon, 20 Nov 2017 12:21:45 +0100 Subject: [erlang-questions] Patch package OTP 20.1.6 released Message-ID: <71f21ae2-e6f5-b179-f287-967309c5d441@ericsson.com> Patch Package: OTP 20.1.6 Git Tag: OTP-20.1.6 Date: 2017-11-20 Trouble Report Id: OTP-14763, OTP-14775, OTP-14778, OTP-14781 Seq num: ERIERL-104, ERIERL-74 System: OTP Release: 20 Application: erts-9.1.5, ssh-4.6.2 Predecessor: OTP 20.1.5 Check out the git tag OTP-20.1.6, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- erts-9.1.5 ------------------------------------------------------ --------------------------------------------------------------------- The erts-9.1.5 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-14775 Application(s): erts Fixed a bug in file closure on Unix; close(2) was retried on EINTR which could cause a different (recently opened) file to be closed as well. OTP-14781 Application(s): erts Related Id(s): OTP-13047 A race-condition when tearing down a connection with active node monitors could cause the runtime system to crash. This bug was introduced in ERTS version 8.0 (OTP 19.0). Full runtime dependencies of erts-9.1.5: kernel-5.0, sasl-3.0.1, stdlib-3.0 --------------------------------------------------------------------- --- ssh-4.6.2 ------------------------------------------------------- --------------------------------------------------------------------- The ssh-4.6.2 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-14763 Application(s): ssh Related Id(s): ERIERL-74 Trailing white space was removed at end of the hello-string. This caused interoperability problems with some other ssh-implementations (e.g OpenSSH 7.3p1 on Solaris 11) OTP-14778 Application(s): ssh Related Id(s): ERIERL-104 Fixes that tcp connections that was immediately closed (SYN, SYNACK, ACK, RST) by a client could be left in a zombie state. Full runtime dependencies of ssh-4.6.2: crypto-3.7.3, erts-6.0, kernel-3.0, public_key-1.4, stdlib-3.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4101 bytes Desc: S/MIME Cryptographic Signature URL: From codewiget95@REDACTED Mon Nov 20 18:03:44 2017 From: codewiget95@REDACTED (code wiget) Date: Mon, 20 Nov 2017 12:03:44 -0500 Subject: [erlang-questions] Tar Without Paths? Message-ID: <527C69B9-AB11-4223-8C97-FF809C299549@gmail.com> Hello, Is it possible to use erl_tar open or create without including the full path to the files? Right now, every add or create adds files with their full path, so when they are extracted, they come in the form of ?/home/wiget/server/files_to_tar/file1.txt instead of files_to_tar/file.txt. The only options in the man pages are irrelevant to my issue, so I assume there is some way around this. If anyone has found a way, I would greatly appreciate your insight! From anthonym@REDACTED Mon Nov 20 18:47:36 2017 From: anthonym@REDACTED (Anthony Molinaro) Date: Mon, 20 Nov 2017 09:47:36 -0800 Subject: [erlang-questions] Tar Without Paths? In-Reply-To: <527C69B9-AB11-4223-8C97-FF809C299549@gmail.com> References: <527C69B9-AB11-4223-8C97-FF809C299549@gmail.com> Message-ID: <0BC6D93D-372D-425D-AE96-00217ABD970E@alumni.caltech.edu> Hi, I haven?t tested it but erl_tar:add/3,4 have a way to specify the NameInTar when adding a file. http://erlang.org/doc/man/erl_tar.html#add-3 HTH, -Anthony > On Nov 20, 2017, at 9:03 AM, code wiget wrote: > > Hello, > > Is it possible to use erl_tar open or create without including the full path to the files? Right now, every add or create adds files with their full path, so when they are extracted, they come in the form of ?/home/wiget/server/files_to_tar/file1.txt instead of files_to_tar/file.txt. The only options in the man pages are irrelevant to my issue, so I assume there is some way around this. If anyone has found a way, I would greatly appreciate your insight! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From fly@REDACTED Mon Nov 20 18:58:44 2017 From: fly@REDACTED (Fred Youhanaie) Date: Mon, 20 Nov 2017 17:58:44 +0000 Subject: [erlang-questions] Tar Without Paths? In-Reply-To: <527C69B9-AB11-4223-8C97-FF809C299549@gmail.com> References: <527C69B9-AB11-4223-8C97-FF809C299549@gmail.com> Message-ID: Hi Are the filenames in the FileList that you supply to create/2 (or /3) in absolute or relative form? I believe erl_tar will use whatever filenames you give it. Cheers, f. On 20/11/17 17:03, code wiget wrote: > Hello, > > Is it possible to use erl_tar open or create without including the full path to the files? Right now, every add or create adds files with their full path, so when they are extracted, they come in the form of ?/home/wiget/server/files_to_tar/file1.txt instead of files_to_tar/file.txt. The only options in the man pages are irrelevant to my issue, so I assume there is some way around this. If anyone has found a way, I would greatly appreciate your insight! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > Fred Youhanaie From raugerfacebook@REDACTED Mon Nov 20 19:18:23 2017 From: raugerfacebook@REDACTED (Ryan Auger) Date: Mon, 20 Nov 2017 13:18:23 -0500 Subject: [erlang-questions] Tar Without Paths? In-Reply-To: References: <527C69B9-AB11-4223-8C97-FF809C299549@gmail.com> Message-ID: <1991380A-6153-427D-9E3A-0422172847F9@gmail.com> Anthony?s solution worked, I can?t believe I missed it. Thanks! > On Nov 20, 2017, at 12:58 PM, Fred Youhanaie wrote: > > Hi > > Are the filenames in the FileList that you supply to create/2 (or /3) in absolute or relative form? > > I believe erl_tar will use whatever filenames you give it. > > Cheers, > f. > > On 20/11/17 17:03, code wiget wrote: >> Hello, >> Is it possible to use erl_tar open or create without including the full path to the files? Right now, every add or create adds files with their full path, so when they are extracted, they come in the form of ?/home/wiget/server/files_to_tar/file1.txt instead of files_to_tar/file.txt. The only options in the man pages are irrelevant to my issue, so I assume there is some way around this. If anyone has found a way, I would greatly appreciate your insight! >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > Fred Youhanaie > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ilya.khaprov@REDACTED Tue Nov 21 11:03:31 2017 From: ilya.khaprov@REDACTED (Ilya Khaprov) Date: Tue, 21 Nov 2017 10:03:31 +0000 Subject: [erlang-questions] Removing eterm from NifEnv. In-Reply-To: References: , Message-ID: Thanks! I think I can group my terms to reduce allocations/garbage collections. Best regards, Ilya ________________________________ From: Lukas Larsson Sent: Monday, November 20, 2017 10:05:15 AM To: Ilya Khaprov Cc: Erlang Questions Subject: Re: [erlang-questions] Removing eterm from NifEnv. Hello, On Sat, Nov 18, 2017 at 8:20 PM, Ilya Khaprov > wrote: I copy (enif_make_copy) an eterm to private NifEnv inside Nif. Later I want to remove that eterm from the NifEnv (actually I want to free memory, but I guess this means removing from NifEnv?s heap). How to do this? I know about enif_free_env, but this function clear entire env. It is not possible to free an individual term from a nif environment. If you want to have that kind of granularity, you have to create one environment per term that you want to individually collect. If the allocation cost of a new env is too large for your application, you could pool env's and use enif_clear_env to re-cycle env's into the pool. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From matwey.kornilov@REDACTED Tue Nov 21 14:02:51 2017 From: matwey.kornilov@REDACTED (Matwey V. Kornilov) Date: Tue, 21 Nov 2017 16:02:51 +0300 Subject: [erlang-questions] Patch package OTP 18.3.4.6 released In-Reply-To: <2ec5ffaa-ce9c-3ad8-9940-e62b27044957@ericsson.com> References: <2ec5ffaa-ce9c-3ad8-9940-e62b27044957@ericsson.com> Message-ID: Hi, By the way, whats about http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-10253 I haven't found any mentions that it is fixed in 18.x. On 16.11.2017 14:29, Hans Nilsson R wrote: > Patch Package: OTP 18.3.4.6 > Git Tag: OTP-18.3.4.6 > Date: 2017-11-16 > Trouble Report Id: OTP-14491, OTP-14514, OTP-14522, OTP-14548, > OTP-14590, OTP-14763, OTP-14765 > Seq num: ERIERL-48, ERIERL-74, ERL-468, ERL-474 > System: OTP > Release: 18 > Application: compiler-6.0.3.1, eldap-1.2.1.1, > erts-7.3.1.4, ssh-4.2.2.4 > Predecessor: OTP 18.3.4.5 > > Check out the git tag OTP-18.3.4.6, and build a full OTP system > including documentation. Apply one or more applications from this > build as patches to your installation using the 'otp_patch_apply' > tool. For information on install requirements, see descriptions for > each application version below. > > --------------------------------------------------------------------- > --- compiler-6.0.3.1 ------------------------------------------------ > --------------------------------------------------------------------- > > The compiler-6.0.3.1 application can be applied independently of > other applications on a full OTP 18 installation. > > --- Fixed Bugs and Malfunctions --- > > OTP-14522 Application(s): compiler > Related Id(s): ERIERL-48 > > Fail labels on guard BIFs weren't taken into account > during an optimization pass, and a bug in the > validation pass sometimes prevented this from being > noticed when a fault occurred. > > > Full runtime dependencies of compiler-6.0.3.1: crypto-3.6, erts-7.0, > hipe-3.12, kernel-4.0, stdlib-2.5 > > > --------------------------------------------------------------------- > --- eldap-1.2.1.1 --------------------------------------------------- > --------------------------------------------------------------------- > > The eldap-1.2.1.1 application can be applied independently of other > applications on a full OTP 18 installation. > > --- Improvements and New Features --- > > OTP-14765 Application(s): eldap, jinterface > > Misc building environment updates > > > Full runtime dependencies of eldap-1.2.1.1: asn1-3.0, erts-6.0, > kernel-3.0, ssl-5.3.4, stdlib-2.0 > > > --------------------------------------------------------------------- > --- erts-7.3.1.4 ---------------------------------------------------- > --------------------------------------------------------------------- > > The erts-7.3.1.4 application can be applied independently of other > applications on a full OTP 18 installation. > > --- Fixed Bugs and Malfunctions --- > > OTP-14491 Application(s): erts > > Fix performance bug in pre-allocators that could cause > them to permanently fall back on normal more expensive > memory allocation. Pre-allocators are used for quick > allocation of short lived meta data used by messages > and other scheduled tasks. Bug exists since OTP_R15B02. > > > OTP-14514 Application(s): erts > > Fixed bug in operator bxor causing erroneuos result > when one operand is a big *negative* integer with the > lowest N*W bits as zero and the other operand not > larger than N*W bits. N is an integer of 1 or larger > and W is 32 or 64 depending on word size. > > > OTP-14548 Application(s): erts > Related Id(s): ERL-468, OTP-11997 > > A timer internal bit-field used for storing scheduler > id was too small. As a result, VM internal timer data > structures could become inconsistent when using 1024 > schedulers on the system. Note that systems with less > than 1024 schedulers are not effected by this bug. > > This bug was introduced in ERTS version 7.0 (OTP 18.0). > > > OTP-14590 Application(s): erts > Related Id(s): ERL-474 > > Fixed bug in binary_to_term and binary_to_atom that > could cause VM crash. Typically happens when the last > character of an UTF8 string is in the range 128 to 255, > but truncated to only one byte. Bug exists in > binary_to_term since ERTS version 5.10.2 (OTP_R16B01) > and binary_to_atom since ERTS version 9.0 (OTP-20.0). > > > Full runtime dependencies of erts-7.3.1.4: kernel-4.0, sasl-2.4, > stdlib-2.5 > > > --------------------------------------------------------------------- > --- ssh-4.2.2.4 ----------------------------------------------------- > --------------------------------------------------------------------- > > Note! The ssh-4.2.2.4 application can *not* be applied independently > of other applications on an arbitrary OTP 18 installation. > > On a full OTP 18 installation, also the following runtime > dependency has to be satisfied: > -- crypto-3.6.3.1 (first satisfied in OTP 18.3.4.5) > > > --- Fixed Bugs and Malfunctions --- > > OTP-14763 Application(s): ssh > Related Id(s): ERIERL-74 > > Trailing white space was removed at end of the > hello-string. This caused interoperability problems > with some other ssh-implementations (e.g OpenSSH 7.3p1 > on Solaris 11) > > > Full runtime dependencies of ssh-4.2.2.4: crypto-3.6.3.1, erts-6.0, > kernel-3.0, public_key-0.22, stdlib-2.3 > > > --------------------------------------------------------------------- > --------------------------------------------------------------------- > --------------------------------------------------------------------- > > > > Mailto: erlang-questions@REDACTED > > Subject: Patch package OTP 18.3.4.6 released > > Body: The OTP 18.3.4.6_open_src.README text > > > > * > * Verify that the git tag OTP-18.3.4.6 has been pushed. > * > > > NOTE: YOU WILL HAVE TO EXPORT THE WEBPAGES: > exit the otp user shell > cd /usr/local/otp/releases/PATCHES/ && gmake > cd ~erlang/www/erlang.se/scripts && ./update_erlang.se PATCHES > (you need passwd for user 'otpwww') > > > From lukas@REDACTED Tue Nov 21 15:22:46 2017 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 21 Nov 2017 15:22:46 +0100 Subject: [erlang-questions] Patch package OTP 18.3.4.6 released In-Reply-To: References: <2ec5ffaa-ce9c-3ad8-9940-e62b27044957@ericsson.com> Message-ID: Hello, On Tue, Nov 21, 2017 at 2:02 PM, Matwey V. Kornilov < matwey.kornilov@REDACTED> wrote: > Hi, > > By the way, whats about > http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-10253 > > I haven't found any mentions that it is fixed in 18.x. > > It is fixed in OTP 20.0 as the change required backwards-incompatible changes to the re library. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz@REDACTED Tue Nov 21 16:23:36 2017 From: aschultz@REDACTED (Andreas Schultz) Date: Tue, 21 Nov 2017 16:23:36 +0100 (CET) Subject: [erlang-questions] C-Node breakage with git master Message-ID: <767319448.3087859.1511277816985.JavaMail.zimbra@tpip.net> Hi, Something in the C-node handling changed between otp-20.1.3 and today git master branch. I have a C-node that implements the remote end of net_adm:ping/1. To do that, it needs to answer to gen:call({net_kernel, Node}, '$gen_call', {is_auth, node()}, infinity) Code is at [1]. In 20.1.3, the call is send as a plain message. With today's git master it first attempts to setup a monitor (a ERL_MONITOR_P message is sent). Monitors are not supported by C-node and ei_xreceive_msg therefor returns an error, causing a connection abort. Reading through some the distribution code and gen.erl code, it seems that the C-node support is somewhat broken to begin with. There is a comment in gen.erl [2] that suggests that a attempting to setup a monitor on a C-node should return with an error. However, this would need to be done on sending side. ei_xreceive_msg has no support to deal with it and the error code would not allow the consumer to implement proper handling. There are two flags in the distribution protocol (DFLAG_DIST_MONITOR and DFLAG_DIST_MONITOR_NAME). C-Nodes do set the DFLAG_DIST_MONITOR, but not the DFLAG_DIST_MONITOR_NAME flag. This seems to be wrong, IMHO the DFLAG_DIST_MONITOR should be cleared. But, I also can't find the place where erlang:monitor/1 would actually check and honor those flags. The monitor BIF seems to always send a monitor request regardless of the node flags. So my questions are: 1. Has anyone an idea what changed to cause the ping/is_auth change? 2. What is the correct way to implement a gen_server in a C-Node or is it in deed currently not possible/broken? Regards Andreas [1]: https://github.com/travelping/capwap-dp/blob/master/src/capwap-dp.c#L943 [2]: https://github.com/erlang/otp/blob/master/lib/stdlib/src/gen.erl#L184 -- Dipl.-Inform. Andreas Schultz email: as@REDACTED phone: +49-391-819099-224 ----------------------- enabling your networks ---------------------- Travelping GmbH phone: +49-391-81 90 99 0 Roentgenstr. 13 fax: +49-391-81 90 99 299 39108 Magdeburg email: info@REDACTED GERMANY web: http://www.travelping.com Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 --------------------------------------------------------------------- From aschultz@REDACTED Tue Nov 21 16:31:38 2017 From: aschultz@REDACTED (Andreas Schultz) Date: Tue, 21 Nov 2017 16:31:38 +0100 (CET) Subject: [erlang-questions] C-Node breakage with git master In-Reply-To: <767319448.3087859.1511277816985.JavaMail.zimbra@tpip.net> References: <767319448.3087859.1511277816985.JavaMail.zimbra@tpip.net> Message-ID: <87813715.3087874.1511278298477.JavaMail.zimbra@tpip.net> ----- On Nov 21, 2017, at 4:23 PM, Andreas Schultz aschultz@REDACTED wrote: > Hi, > > Something in the C-node handling changed between otp-20.1.3 and > today git master branch. Did sent to early, the behavior was changed by this commit: https://github.com/erlang/otp/commit/17e198d6ee60f7dec9abfed272cf4226aea44535 I think the removal of the DFLAG_DIST_MONITOR and DFLAG_DIST_MONITOR_NAME in that cset is a mistake. Also, the removed DFLAG_DIST_MONITOR test proves IMHO that a C-Node should not set that flag. Comments? Regards Andreas > > I have a C-node that implements the remote end of net_adm:ping/1. > To do that, it needs to answer to > > gen:call({net_kernel, Node}, '$gen_call', {is_auth, node()}, infinity) > > Code is at [1]. > > In 20.1.3, the call is send as a plain message. With today's git master > it first attempts to setup a monitor (a ERL_MONITOR_P message is sent). > > Monitors are not supported by C-node and ei_xreceive_msg therefor returns > an error, causing a connection abort. > > Reading through some the distribution code and gen.erl code, it seems > that the C-node support is somewhat broken to begin with. There is > a comment in gen.erl [2] that suggests that a attempting to setup a > monitor on a C-node should return with an error. However, this would > need to be done on sending side. ei_xreceive_msg has no support to > deal with it and the error code would not allow the consumer to > implement proper handling. > > There are two flags in the distribution protocol (DFLAG_DIST_MONITOR and > DFLAG_DIST_MONITOR_NAME). C-Nodes do set the DFLAG_DIST_MONITOR, but not > the DFLAG_DIST_MONITOR_NAME flag. This seems to be wrong, IMHO the > DFLAG_DIST_MONITOR should be cleared. > > But, I also can't find the place where erlang:monitor/1 would actually > check and honor those flags. The monitor BIF seems to always send a > monitor request regardless of the node flags. > > So my questions are: > > 1. Has anyone an idea what changed to cause the ping/is_auth change? > 2. What is the correct way to implement a gen_server in a C-Node or > is it in deed currently not possible/broken? > > Regards > Andreas > > [1]: https://github.com/travelping/capwap-dp/blob/master/src/capwap-dp.c#L943 > [2]: https://github.com/erlang/otp/blob/master/lib/stdlib/src/gen.erl#L184 > -- > Dipl.-Inform. Andreas Schultz > > email: as@REDACTED > phone: +49-391-819099-224 > > ----------------------- enabling your networks ---------------------- > > Travelping GmbH phone: +49-391-81 90 99 0 > Roentgenstr. 13 fax: +49-391-81 90 99 299 > 39108 Magdeburg email: info@REDACTED > GERMANY web: http://www.travelping.com > > Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 > Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 > --------------------------------------------------------------------- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sverker.eriksson@REDACTED Tue Nov 21 18:19:37 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Tue, 21 Nov 2017 17:19:37 +0000 Subject: [erlang-questions] C-Node breakage with git master In-Reply-To: <87813715.3087874.1511278298477.JavaMail.zimbra@tpip.net> References: <767319448.3087859.1511277816985.JavaMail.zimbra@tpip.net> <87813715.3087874.1511278298477.JavaMail.zimbra@tpip.net> Message-ID: <1511284777.15708.52.camel@ericsson.com> Hi Andreas Current master is broken with respect to monitor of c-node "processes". I'm currently schratching my head trying to fix this mess. Nice to have a shoulder to cry on. The commit?https://github.com/erlang/otp/commit/17e198d6ee60f7dec9abfed 272cf4226?is part of a work to make distributed operations (like send, monitor, link, etc) to *not* wait for the connections setup to complete. They should just enqueue their request and trigger a connection setup to commence. This can be done in a backward compatible manner as all of them have a truly asynchronous interface. All except erlang:monitor, when called toward c-nodes which do not support process monitoring. Old behaviour is to throw badarg. But that would force erlang:monitor to still be synchronous waiting for connection setup in order to know if the node supports it or not. So the idea is to slightly change the behavoir and instead of badarg go ahead and create the monitor but only let it supervise the connection. That is, you will only get 'DOWN' message with 'noconnection' from such a monitor. This is similar to what gen:call implements today by cathing badarg and using monitor_node. Yes, erl_interface sets DFLAG_DIST_MONITOR but it does not implement it. Why? I think it's an old sloppy fix to make?erl_global_register() work, where the c-node receives a monitor request as part of the reply which it just ignores. /Sverker On tis, 2017-11-21 at 16:31 +0100, Andreas Schultz wrote: > ----- On Nov 21, 2017, at 4:23 PM, Andreas Schultz aschultz@REDACTED > wrote: > > > > > Hi, > > > > Something in the C-node handling changed between otp-20.1.3 and > > today git master branch. > Did sent to early, the behavior was changed by this commit: > > https://github.com/erlang/otp/commit/17e198d6ee60f7dec9abfed272cf4226 > aea44535 > > I think the removal of the DFLAG_DIST_MONITOR and > DFLAG_DIST_MONITOR_NAME > in that cset is a mistake. > > Also, the removed DFLAG_DIST_MONITOR test proves IMHO that a > C-Node should not set that flag. > > Comments? > > Regards > Andreas > > > > > > > I have a C-node that implements the remote end of net_adm:ping/1. > > To do that, it needs to answer to > > > > ?gen:call({net_kernel, Node}, '$gen_call', {is_auth, node()}, > > infinity) > > > > Code is at [1]. > > > > In 20.1.3, the call is send as a plain message. With today's git > > master > > it first attempts to setup a monitor (a ERL_MONITOR_P message is > > sent). > > > > Monitors are not supported by C-node and ei_xreceive_msg therefor > > returns > > an error, causing a connection abort. > > > > Reading through some the distribution code and gen.erl code, it > > seems > > that the C-node support is somewhat broken to begin with. There is > > a comment in gen.erl [2] that suggests that a attempting to setup a > > monitor on a C-node should return with an error. However, this > > would > > need to be done on sending side. ei_xreceive_msg has no support to > > deal with it and the error code would not allow the consumer to > > implement proper handling. > > > > There are two flags in the distribution protocol > > (DFLAG_DIST_MONITOR and > > DFLAG_DIST_MONITOR_NAME). C-Nodes do set the DFLAG_DIST_MONITOR, > > but not > > the DFLAG_DIST_MONITOR_NAME flag. This seems to be wrong, IMHO the > > DFLAG_DIST_MONITOR should be cleared. > > > > But, I also can't find the place where erlang:monitor/1 would > > actually > > check and honor those flags. The monitor BIF seems to always send a > > monitor request regardless of the node flags. > > > > So my questions are: > > > > 1. Has anyone an idea what changed to cause the ping/is_auth > > change? > > 2. What is the correct way to implement a gen_server in a C-Node or > > ? is it in deed currently not possible/broken? > > > > Regards > > Andreas > > > > [1]: https://github.com/travelping/capwap-dp/blob/master/src/capwap > > -dp.c#L943 > > [2]: https://github.com/erlang/otp/blob/master/lib/stdlib/src/gen.e > > rl#L184 > > -- > > Dipl.-Inform. Andreas Schultz > > > > email: as@REDACTED > > phone: +49-391-819099-224 > > > > ----------------------- enabling your networks -------------------- > > -- > > > > Travelping GmbH?????????????????????phone:??+49-391-81 90 99 0 > > Roentgenstr.??13????????????????????fax:????+49-391-81 90 99 299 > > 39108 Magdeburg?????????????????????email:??info@REDACTED > > GERMANY?????????????????????????????web:????http://www.travelping.c > > om > > > > Company Registration: Amtsgericht Stendal????????Reg No.:???HRB > > 10578 > > Geschaeftsfuehrer: Holger Winkelmann??????????VAT ID No.: > > DE236673780 > > ----------------------------------------------------------------- > > ---- > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions From aschultz@REDACTED Tue Nov 21 18:31:04 2017 From: aschultz@REDACTED (Andreas Schultz) Date: Tue, 21 Nov 2017 18:31:04 +0100 (CET) Subject: [erlang-questions] C-Node breakage with git master In-Reply-To: <1511284777.15708.52.camel@ericsson.com> References: <767319448.3087859.1511277816985.JavaMail.zimbra@tpip.net> <87813715.3087874.1511278298477.JavaMail.zimbra@tpip.net> <1511284777.15708.52.camel@ericsson.com> Message-ID: <1872734591.3088063.1511285464784.JavaMail.zimbra@tpip.net> Hi, ----- On Nov 21, 2017, at 6:19 PM, Sverker Eriksson sverker.eriksson@REDACTED wrote: > Hi Andreas > > Current master is broken with respect to monitor of c-node "processes". Good to know ;-) > I'm currently schratching my head trying to fix this mess. Nice to have > a shoulder to cry on. > > The commit?https://github.com/erlang/otp/commit/17e198d6ee60f7dec9abfed > 272cf4226?is part of a work to make distributed operations (like send, > monitor, link, etc) to *not* wait for the connections setup to > complete. They should just enqueue their request and trigger a > connection setup to commence. > > This can be done in a backward compatible manner as all of them have a > truly asynchronous interface. > > All except erlang:monitor, when called toward c-nodes which do not > support process monitoring. Old behaviour is to throw badarg. But that > would force erlang:monitor to still be synchronous waiting for > connection setup in order to know if the node supports it or not. I would be nice if we could have to way to support monitors in a C-node as well. It would have to be the responsibility of the C-node to implement whatever kind of monitoring it sees fit when such a request comes in. And since I'm dreaming, cached atoms in C-nodes would be great ;-) > So the idea is to slightly change the behavoir and instead of badarg go > ahead and create the monitor but only let it supervise the connection. > That is, you will only get 'DOWN' message with 'noconnection' from such > a monitor. This is similar to what gen:call implements today by cathing > badarg and using monitor_node. Sounds good to me. Andreas > Yes, erl_interface sets DFLAG_DIST_MONITOR but it does not implement > it. Why? I think it's an old sloppy fix to make?erl_global_register() > work, where the c-node receives a monitor request as part of the reply > which it just ignores. > > > > /Sverker > > > > On tis, 2017-11-21 at 16:31 +0100, Andreas Schultz wrote: >> ----- On Nov 21, 2017, at 4:23 PM, Andreas Schultz aschultz@REDACTED >> wrote: >> >> > >> > Hi, >> > >> > Something in the C-node handling changed between otp-20.1.3 and >> > today git master branch. >> Did sent to early, the behavior was changed by this commit: >> >> https://github.com/erlang/otp/commit/17e198d6ee60f7dec9abfed272cf4226 >> aea44535 >> >> I think the removal of the DFLAG_DIST_MONITOR and >> DFLAG_DIST_MONITOR_NAME >> in that cset is a mistake. >> >> Also, the removed DFLAG_DIST_MONITOR test proves IMHO that a >> C-Node should not set that flag. >> >> Comments? >> >> Regards >> Andreas >> >> > >> > >> > I have a C-node that implements the remote end of net_adm:ping/1. >> > To do that, it needs to answer to >> > >> > ?gen:call({net_kernel, Node}, '$gen_call', {is_auth, node()}, >> > infinity) >> > >> > Code is at [1]. >> > >> > In 20.1.3, the call is send as a plain message. With today's git >> > master >> > it first attempts to setup a monitor (a ERL_MONITOR_P message is >> > sent). >> > >> > Monitors are not supported by C-node and ei_xreceive_msg therefor >> > returns >> > an error, causing a connection abort. >> > >> > Reading through some the distribution code and gen.erl code, it >> > seems >> > that the C-node support is somewhat broken to begin with. There is >> > a comment in gen.erl [2] that suggests that a attempting to setup a >> > monitor on a C-node should return with an error. However, this >> > would >> > need to be done on sending side. ei_xreceive_msg has no support to >> > deal with it and the error code would not allow the consumer to >> > implement proper handling. >> > >> > There are two flags in the distribution protocol >> > (DFLAG_DIST_MONITOR and >> > DFLAG_DIST_MONITOR_NAME). C-Nodes do set the DFLAG_DIST_MONITOR, >> > but not >> > the DFLAG_DIST_MONITOR_NAME flag. This seems to be wrong, IMHO the >> > DFLAG_DIST_MONITOR should be cleared. >> > >> > But, I also can't find the place where erlang:monitor/1 would >> > actually >> > check and honor those flags. The monitor BIF seems to always send a >> > monitor request regardless of the node flags. >> > >> > So my questions are: >> > >> > 1. Has anyone an idea what changed to cause the ping/is_auth >> > change? >> > 2. What is the correct way to implement a gen_server in a C-Node or >> > ? is it in deed currently not possible/broken? >> > >> > Regards >> > Andreas >> > >> > [1]: https://github.com/travelping/capwap-dp/blob/master/src/capwap >> > -dp.c#L943 >> > [2]: https://github.com/erlang/otp/blob/master/lib/stdlib/src/gen.e >> > rl#L184 >> > -- >> > Dipl.-Inform. Andreas Schultz >> > >> > email: as@REDACTED >> > phone: +49-391-819099-224 >> > >> > ----------------------- enabling your networks -------------------- >> > -- >> > >> > Travelping GmbH?????????????????????phone:??+49-391-81 90 99 0 >> > Roentgenstr.??13????????????????????fax:????+49-391-81 90 99 299 >> > 39108 Magdeburg?????????????????????email:??info@REDACTED >> > GERMANY?????????????????????????????web:????http://www.travelping.c >> > om >> > >> > Company Registration: Amtsgericht Stendal????????Reg No.:???HRB >> > 10578 >> > Geschaeftsfuehrer: Holger Winkelmann??????????VAT ID No.: >> > DE236673780 >> > ----------------------------------------------------------------- >> > ---- >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions From arunp@REDACTED Wed Nov 22 05:16:23 2017 From: arunp@REDACTED (Arun) Date: Wed, 22 Nov 2017 09:46:23 +0530 Subject: [erlang-questions] Mnesia deleting log file Message-ID: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> Dear all, I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing all the persistent configurations. I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem and how do I fix it? Mnesia('test@REDACTED'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running, 'test@REDACTED'} -------------- next part -------------- An HTML attachment was scrubbed... URL: From dgud@REDACTED Wed Nov 22 09:25:50 2017 From: dgud@REDACTED (Dan Gudmundsson) Date: Wed, 22 Nov 2017 08:25:50 +0000 Subject: [erlang-questions] Mnesia deleting log file In-Reply-To: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> References: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> Message-ID: You should get 'node_not_running' when mnesia is not running (stopped or have crashed) so something is very strange. Don't you get other mnesia log entries? Try to enable debug printouts with mnesia:start([{debug, debug}]). or erl -mnesia debug debug /Dan On Wed, Nov 22, 2017 at 5:16 AM Arun wrote: > Dear all, > > I've a program written in erlang which uses Mnesia application as the > database application. I've a table created by name "configuration_table" > which stores > > certain configurations that need to be persistent. Occasionally, whenever > I restart my program the following error is thrown by mnesia and I end up > losing > > all the persistent configurations. > > I've searched about this problem in the erlang documentation and all it > tells is "Node not running". What could be the probable cause for this > problem > > and how do I fix it? > > Mnesia('test@REDACTED '): Data may be missing, > Corrupt logfile deleted: > "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running, > 'test@REDACTED '} > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jinni.park@REDACTED Wed Nov 22 10:09:21 2017 From: jinni.park@REDACTED (Park, Sungjin) Date: Wed, 22 Nov 2017 18:09:21 +0900 Subject: [erlang-questions] Non-handled tls packets in the message queue Message-ID: I encountered a problem in receiving tls packets in my application. After some investigation, I figured out that the process has a tls packet in it's message queue but didn't have a chance to handle it. This happens in a specific situation when the server closes the connection after sending large data which is split into several packets so that some of them remain in the message queue when the socket closes by FIN. I followed OTP code and applied a patch below and confirmed it solved the problem. But I'm not quite sure if this is the right way to solve the problem nor why it was written that way in the first place -- dtls_connection seems to be working just as my patch. --- a/lib/ssl/src/tls_connection.erl +++ b/lib/ssl/src/tls_connection.erl @@ -602,12 +602,8 @@ next_record(#state{protocol_buffers = next_record(#state{protocol_buffers = #protocol_buffers{tls_packets = [], tls_cipher_texts = []}, socket = Socket, transport_cb = Transport} = State) -> - case tls_socket:setopts(Transport, Socket, [{active,once}]) of - ok -> - {no_record, State}; - _ -> - {socket_closed, State} - end; + tls_socket:setopts(Transport, Socket, [{active,once}]), + {no_record, State}; next_record(State) -> {no_record, State}. -- Park, Sungjin -------------- next part -------------- An HTML attachment was scrubbed... URL: From Michael.K.Schmidt@REDACTED Wed Nov 22 14:21:57 2017 From: Michael.K.Schmidt@REDACTED (Michael Schmidt) Date: Wed, 22 Nov 2017 13:21:57 +0000 Subject: [erlang-questions] Mnesia deleting log file In-Reply-To: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> References: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> Message-ID: Hi Arun, If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data). We fight this issue on embedded devices a lot. Look ?Configuration Parameters? section here: http://erlang.org/doc/man/mnesia.html#id69584 You will want to set dump_log_update_in_place to false. This will make it write out the table to a new file, and rename it the very end. One other option to look at is dump_log_time_threshold to change how often items are flushed to disk. For configuration data, you can likely set this to a few seconds. These options can be set via the sys.config file as well Mike From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Arun Sent: Tuesday, November 21, 2017 10:16 PM To: erlang-questions@REDACTED Subject: [erlang-questions] Mnesia deleting log file Dear all, I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing all the persistent configurations. I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem and how do I fix it? Mnesia('test@REDACTED'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running, 'test@REDACTED'} ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Nov 22 14:54:11 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 22 Nov 2017 13:54:11 +0000 Subject: [erlang-questions] Mnesia deleting log file In-Reply-To: References: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> Message-ID: This thread makes we wonder about the following section in the Mnesia documentation: "If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority." In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should. With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question. [1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole. On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt < Michael.K.Schmidt@REDACTED> wrote: > Hi Arun, > > > > If you happen to reboot when Mnesia is writing out its logfile, it can > become corrupt (and lose data). We fight this issue on embedded devices a > lot. > > > > Look ?Configuration Parameters? section here: > > http://erlang.org/doc/man/mnesia.html#id69584 > > > > You will want to set dump_log_update_in_place to false. This will make > it write out the table to a new file, and rename it the very end. > > > > One other option to look at is dump_log_time_threshold to change how > often items are flushed to disk. For configuration data, you can likely > set this to a few seconds. > > > > These options can be set via the sys.config file as well > > > > Mike > > > > *From:* erlang-questions-bounces@REDACTED [mailto: > erlang-questions-bounces@REDACTED] *On Behalf Of *Arun > *Sent:* Tuesday, November 21, 2017 10:16 PM > *To:* erlang-questions@REDACTED > *Subject:* [erlang-questions] Mnesia deleting log file > > > > Dear all, > > I've a program written in erlang which uses Mnesia application as the > database application. I've a table created by name "configuration_table" > which stores > > certain configurations that need to be persistent. Occasionally, whenever > I restart my program the following error is thrown by mnesia and I end up > losing > > all the persistent configurations. > > I've searched about this problem in the erlang documentation and all it > tells is "Node not running". What could be the probable cause for this > problem > > and how do I fix it? > > Mnesia('test@REDACTED '): Data may be missing, > Corrupt logfile deleted: > "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running, > 'test@REDACTED '} > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > ______________________________________________________________________ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Nov 22 20:43:10 2017 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 22 Nov 2017 14:43:10 -0500 (EST) Subject: [erlang-questions] Strings - deprecated functions Message-ID: <1511379790.16757707@apps.rackspace.com> Dear Gods of Erlang, "This module has been reworked in Erlang/OTP 20 to handle [ unicode:chardata() ]( http://erlang.org/doc/man/unicode.html#type-chardata ) and operate on grapheme clusters. The [ old functions ]( http://erlang.org/doc/man/string.html#oldapi ) that only work on Latin-1 lists as input are still available but should not be used. They will be deprecated in Erlang/OTP 21." I'm sorry. I've brought up this issue before and got lots of push back. But every time I look up tried and true and long-used string functions to find that they are deprecated and will be dropped in future Erlang releases my blood pressure soars. Both my wife and my doctor tell me that at my age this is a dangerous thing. I do understand the importance and necessity of Unicode. And applaud the addition of Unicode functions. But the deprecated string functions have a long history. The English language and Latin-1 characters are widely used around the world. Yes, it should be easy for programmers to translate code from one user language to another. But I'm not convinced that the Gods of Erlang have found the optimal solution by dropping all Latin-1 string functions. My particular application is directed toward English speakers. So, until further notice, I have no use for Unicode. I don't want to sound like nationalist pig, but I think dropping the Latin-1 string functions from future Erlang releases is a BIG mistake. I look up tokens/2, a function that I use fairly frequently, and I see that it's deprecated. I look up the suggested replacement and I see lexemes/2. So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I see that a lexeme is "a meaningful linguistic unit." Meaning what? I just want to turn "this and that" into "This And That." I read further in the Erlang docs and I see "grapheme cluster." WHAT THE ... IS GRAPHEME CLUSTER? I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a unit of a writing system." Ah yes, grapheme is defined in the docs. But I have to read and re-read the definition to understand what the God's of Erlang mean by a "graphene cluster." And I'm still not sure I get it. It sounds like someone took a linguistics class and is trying to show off. But now I've spent 30 minutes--- time that I don't have to waste trying to figure out how do a simple manipulation of "this and that." Recurse the next time I want to look up a string function in the Erlang docs. SOLUTION Keep the Latin-1 string functions. Put them in a separate library if necessary. Or put the new Unicode functions in a separate library. But don't arbitrarily drop them. Some folks have suggested that I maintain my own library of the deprecated Latin1 functions. But why should I have to do that? How does that help other folks with the same issue? Bottom line: please please please do not drop the existing Latin-1 string functions. Please don't. Best wishes, LRP -------------- next part -------------- An HTML attachment was scrubbed... URL: From list1@REDACTED Wed Nov 22 20:59:45 2017 From: list1@REDACTED (Grzegorz Junka) Date: Wed, 22 Nov 2017 19:59:45 +0000 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511379790.16757707@apps.rackspace.com> References: <1511379790.16757707@apps.rackspace.com> Message-ID: <4402c210-5e88-b0ce-231f-1d3657fcdce8@gjunka.com> Dear Lloyd, Isn't this more about documentation than the code? What I am reading is that you want to keep the old functions because you don't understand how the new functions work. Shouldn't you rather ask for a more clear documentation? Is there anything in the old functions that is not supported in the new functions? GrzegorzJ On 22/11/2017 19:43, lloyd@REDACTED wrote: > > Dear Gods of Erlang, > > "This module has been reworked in Erlang/OTP 20 to handle > unicode:chardata() > and operate on > grapheme clusters. The old functions > that only work on > Latin-1 lists as input are still available but should not be used. > They will be deprecated in Erlang/OTP 21." > > I'm sorry. I've brought up this issue before and got lots of push back. > > But every time I look up tried and true and long-used string functions > to find that they are deprecated and will be dropped in future Erlang > releases my blood pressure soars. Both my wife and my doctor tell me > that at my age this is a dangerous thing. > > I do understand the importance and necessity of Unicode. And applaud > the addition of Unicode functions. > > But the deprecated string functions have a long history. The English > language and Latin-1 characters are widely used around the world. > > Yes, it should be easy for programmers to translate code from one user > language to another. But I'm not convinced that the Gods of Erlang > have found the optimal solution by dropping all Latin-1 string functions. > > My particular application is directed toward English speakers. So, > until further notice, I have no use for Unicode. > > I don't want to sound like nationalist pig, but I think dropping the > Latin-1 string functions from future Erlang releases is a BIG mistake. > > I look up tokens/2, a function that I use fairly frequently, and I see > that it's deprecated. I look up the suggested replacement and I see > lexemes/2. > > So I ask, what the ... is a lexeme? I look it up in Merriam-Webster > and I see that a lexeme is? "a meaningful linguistic unit." > > Meaning what? I just want to turn "this and that" into "This And That." > > I read further in the Erlang docs and I see "grapheme cluster."? WHAT > THE ... IS GRAPHEME CLUSTER? > > I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a > unit of a writing system." > > Ah yes, grapheme is defined in the docs. But I have to read and > re-read the definition to understand what the God's of Erlang mean by > a "graphene cluster." And I'm still not sure I get it. > > It sounds like someone took a linguistics class and is trying to show off. > > But now I've spent 30 minutes--- time that I don't have to waste > trying to figure out how do a simple manipulation of "this and that." > Recurse the next time I want to look up a string function in the > Erlang docs. > > SOLUTION > > Keep the Latin-1 string functions. Put them in a separate library if > necessary. Or put the new Unicode functions in a separate library. But > don't arbitrarily drop them. > > Some folks have suggested that I maintain my own library of the > deprecated Latin1 functions. But why should I have to do that? How > does that help other folks with the same issue? > > Bottom line: please please please do not drop the existing Latin-1 > string functions. > > Please don't. > > Best wishes, > > LRP > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Nov 22 21:26:24 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 22 Nov 2017 20:26:24 +0000 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <4402c210-5e88-b0ce-231f-1d3657fcdce8@gjunka.com> References: <1511379790.16757707@apps.rackspace.com> <4402c210-5e88-b0ce-231f-1d3657fcdce8@gjunka.com> Message-ID: In this case, the words comes partly from terms you would find in linguistics, partly words which have specific meaning in the unicode standard. The problem with Latin-1 and ISO8859-1 and ISO8859-15 are that they work somewhat well for Western Latin languages, but it falls short on almost everything else. If your only concern is truly English text, then there should be no worry at all, since that uses ASCII and the predominant Unicode enconding, UTF-8 was chosen such that there is a 1-1 overlap between the first 128 characters and ASCII. However, Unicode imposes some difficulties. The most notable one is that you have several ways of writing symbols such as the danish ? and ?: Either as a specific character, or as a combination: and A and a small ring on top for instance. In languages the written symbols are graphemes, and collections of symbols forming tokens or words are lexemes. However, because one grapheme can be represented as one or several characters, the notion of a grapheme cluster arises: several code-points which form a single grapheme. It is of utmost importance for certain Asian writing systems in which a single grapheme is composed out of several smaller ones. For ASCII, however, string:lexemes/2 would work exactly like string:tokens/2. Yet it will handle far more cases. Unicode presents its own set of complexities. There are several ways of writing a unicode string which is "the same" string in that it renders equally to the human eye. Hence, there are some routines for handling normalization, canonization and collation which by no means are easy to handle. And finally, it would probably be good to define those terms in the documentation. I don't think they are well-known to most people. On Wed, Nov 22, 2017 at 8:59 PM Grzegorz Junka wrote: > Dear Lloyd, > > Isn't this more about documentation than the code? What I am reading is > that you want to keep the old functions because you don't understand how > the new functions work. Shouldn't you rather ask for a more clear > documentation? Is there anything in the old functions that is not supported > in the new functions? > > GrzegorzJ > > On 22/11/2017 19:43, lloyd@REDACTED wrote: > > Dear Gods of Erlang, > > > > "This module has been reworked in Erlang/OTP 20 to handle > unicode:chardata() > and operate on grapheme clusters. The old functions > that only work on Latin-1 > lists as input are still available but should not be used. They will be > deprecated in Erlang/OTP 21." > > > > I'm sorry. I've brought up this issue before and got lots of push back. > > > > But every time I look up tried and true and long-used string functions to > find that they are deprecated and will be dropped in future Erlang releases > my blood pressure soars. Both my wife and my doctor tell me that at my age > this is a dangerous thing. > > > > I do understand the importance and necessity of Unicode. And applaud the > addition of Unicode functions. > > > > But the deprecated string functions have a long history. The English > language and Latin-1 characters are widely used around the world. > > > > Yes, it should be easy for programmers to translate code from one user > language to another. But I'm not convinced that the Gods of Erlang have > found the optimal solution by dropping all Latin-1 string functions. > > > > My particular application is directed toward English speakers. So, until > further notice, I have no use for Unicode. > > > > I don't want to sound like nationalist pig, but I think dropping the > Latin-1 string functions from future Erlang releases is a BIG mistake. > > > > I look up tokens/2, a function that I use fairly frequently, and I see > that it's deprecated. I look up the suggested replacement and I see > lexemes/2. > > > > So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I > see that a lexeme is "a meaningful linguistic unit." > > > > Meaning what? I just want to turn "this and that" into "This And That." > > > > I read further in the Erlang docs and I see "grapheme cluster." WHAT THE > ... IS GRAPHEME CLUSTER? > > > > I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a > unit of a writing system." > > > > Ah yes, grapheme is defined in the docs. But I have to read and re-read > the definition to understand what the God's of Erlang mean by a "graphene > cluster." And I'm still not sure I get it. > > > > It sounds like someone took a linguistics class and is trying to show off. > > > > But now I've spent 30 minutes--- time that I don't have to waste trying to > figure out how do a simple manipulation of "this and that." Recurse the > next time I want to look up a string function in the Erlang docs. > > > > SOLUTION > > > > Keep the Latin-1 string functions. Put them in a separate library if > necessary. Or put the new Unicode functions in a separate library. But > don't arbitrarily drop them. > > > > Some folks have suggested that I maintain my own library of the deprecated > Latin1 functions. But why should I have to do that? How does that help > other folks with the same issue? > > > > Bottom line: please please please do not drop the existing Latin-1 string > functions. > > > > Please don't. > > > > Best wishes, > > > > LRP > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Nov 22 21:28:52 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 22 Nov 2017 21:28:52 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511379790.16757707@apps.rackspace.com> References: <1511379790.16757707@apps.rackspace.com> Message-ID: <03e7d789-6487-b2dc-0a5b-698d97078ced@ninenines.eu> Calm down. Considering how ubiquitous the string module is, these functions are not going to be removed for at least a few years. That gives you plenty of time to understand the new string module. Perhaps during this journey you can help make the documentation for the module more user friendly by sending patches or opening tickets at bugs.erlang.org. I'll admit that the current documentation does confuse me personally, though I've not needed to use it yet. Unfortunately languages are complex and Unicode is therefore also complex. There's no real way around that. Even if you target English speakers it's likely that you will need Unicode, because many things require it like names or addresses for example. So even if it feels like you won't need it (and maybe you won't) it's a good idea to be ready for it. I wouldn't say latin1 is widely used anymore. Most of everything uses Unicode nowadays. Nearly everything switched to Unicode, Erlang is one of the last. Even your email was sent encoded in utf-8. On 11/22/2017 08:43 PM, lloyd@REDACTED wrote: > Dear Gods of Erlang, > > "This module has been reworked in Erlang/OTP 20 to handle > unicode:chardata() > and operate on > grapheme clusters. The old functions > that only work on Latin-1 > lists as input are still available but should not be used. They will be > deprecated in Erlang/OTP 21." > > I'm sorry. I've brought up this issue before and got lots of push back. > > But every time I look up tried and true and long-used string functions > to find that they are deprecated and will be dropped in future Erlang > releases my blood pressure soars. Both my wife and my doctor tell me > that at my age this is a dangerous thing. > > I do understand the importance and necessity of Unicode. And applaud the > addition of Unicode functions. > > But the deprecated string functions have a long history. The English > language and Latin-1 characters are widely used around the world. > > Yes, it should be easy for programmers to translate code from one user > language to another. But I'm not convinced that the Gods of Erlang have > found the optimal solution by dropping all Latin-1 string functions. > > My particular application is directed toward English speakers. So, until > further notice, I have no use for Unicode. > > I don't want to sound like nationalist pig, but I think dropping the > Latin-1 string functions from future Erlang releases is a BIG mistake. > > I look up tokens/2, a function that I use fairly frequently, and I see > that it's deprecated. I look up the suggested replacement and I see > lexemes/2. > > So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and > I see that a lexeme is? "a meaningful linguistic unit." > > Meaning what? I just want to turn "this and that" into "This And That." > > I read further in the Erlang docs and I see "grapheme cluster."? WHAT > THE ... IS GRAPHEME CLUSTER? > > I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a > unit of a writing system." > > Ah yes, grapheme is defined in the docs. But I have to read and re-read > the definition to understand what the God's of Erlang mean by a > "graphene cluster." And I'm still not sure I get it. > > It sounds like someone took a linguistics class and is trying to show off. > > But now I've spent 30 minutes--- time that I don't have to waste trying > to figure out how do a simple manipulation of "this and that." Recurse > the next time I want to look up a string function in the Erlang docs. > > SOLUTION > > Keep the Latin-1 string functions. Put them in a separate library if > necessary. Or put the new Unicode functions in a separate library. But > don't arbitrarily drop them. > > Some folks have suggested that I maintain my own library of the > deprecated Latin1 functions. But why should I have to do that? How does > that help other folks with the same issue? > > Bottom line: please please please do not drop the existing Latin-1 > string functions. > > Please don't. > > Best wishes, > > LRP > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From eric.pailleau@REDACTED Wed Nov 22 22:11:32 2017 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Wed, 22 Nov 2017 22:11:32 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511379790.16757707@apps.rackspace.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: DIa_M7LUQAEhYnC.jpg Type: image/jpeg Size: 169380 bytes Desc: not available URL: From Michael.K.Schmidt@REDACTED Wed Nov 22 22:16:05 2017 From: Michael.K.Schmidt@REDACTED (Michael Schmidt) Date: Wed, 22 Nov 2017 21:16:05 +0000 Subject: [erlang-questions] Mnesia deleting log file In-Reply-To: References: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> Message-ID: I agree?setting dump_log_in_place to false makes extremely unlikely for mnesia to fail in this way. Both JFFS2 and UBIFS ensure that rename is atomic, even when power is pulled. For desktop applications, the culprit is likely the VM being killed, not the whole system crashing. Thus the rename was either executed or not; either way we have a complete DAT file. There is always a chance of things like this happening: http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but this shouldn?t happen on most hardware. Note that the default for dump_log_time_threshold is 3 minutes. Speed vs Safety is always a trade-off, but many people will be surprised to lose data in a crash 2 minutes after saving. From: Jesper Louis Andersen [mailto:jesper.louis.andersen@REDACTED] Sent: Wednesday, November 22, 2017 7:54 AM To: Michael Schmidt Cc: Arun ; erlang-questions@REDACTED Subject: Re: [erlang-questions] Mnesia deleting log file This thread makes we wonder about the following section in the Mnesia documentation: "If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority." In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should. With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question. [1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole. On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt > wrote: Hi Arun, If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data). We fight this issue on embedded devices a lot. Look ?Configuration Parameters? section here: http://erlang.org/doc/man/mnesia.html#id69584 You will want to set dump_log_update_in_place to false. This will make it write out the table to a new file, and rename it the very end. One other option to look at is dump_log_time_threshold to change how often items are flushed to disk. For configuration data, you can likely set this to a few seconds. These options can be set via the sys.config file as well Mike From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Arun Sent: Tuesday, November 21, 2017 10:16 PM To: erlang-questions@REDACTED Subject: [erlang-questions] Mnesia deleting log file Dear all, I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing all the persistent configurations. I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem and how do I fix it? Mnesia('test@REDACTED'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running, 'test@REDACTED'} ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________ _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Catenacci@REDACTED Wed Nov 22 22:43:04 2017 From: Catenacci@REDACTED (Onorio Catenacci) Date: Wed, 22 Nov 2017 16:43:04 -0500 Subject: [erlang-questions] Strings - deprecated functions Message-ID: I have had several colleagues apply analogous logic in regards to concurrency and the actor model as you've applied to using Latin-1. That is colleagues tell me that they will never need concurrency and their apps will never be run on multiple machines so why pay the "mental tax" of using Erlang? I'm not saying you're wrong but maybe you want to keep your options open. -- Onorio Catenacci http://onor.io http://www.google.com/+OnorioCatenacci -------------- next part -------------- An HTML attachment was scrubbed... URL: From dgud@REDACTED Wed Nov 22 22:45:25 2017 From: dgud@REDACTED (Dan Gudmundsson) Date: Wed, 22 Nov 2017 21:45:25 +0000 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <4402c210-5e88-b0ce-231f-1d3657fcdce8@gjunka.com> References: <1511379790.16757707@apps.rackspace.com> <4402c210-5e88-b0ce-231f-1d3657fcdce8@gjunka.com> Message-ID: On Wed, Nov 22, 2017 at 8:59 PM Grzegorz Junka wrote: > Dear Lloyd, > > Isn't this more about documentation than the code? What I am reading is > that you want to keep the old functions because you don't understand how > the new functions work. Shouldn't you rather ask for a more clear > documentation? Is there anything in the old functions that is not supported > in the new functions? > > GrzegorzJ > > On 22/11/2017 19:43, lloyd@REDACTED wrote: > > Dear Gods of Erlang, > > > > "This module has been reworked in Erlang/OTP 20 to handle > unicode:chardata() > and operate on grapheme clusters. The old functions > that only work on Latin-1 > lists as input are still available but should not be used. They will be > deprecated in Erlang/OTP 21." > > > > The new functions also works on binaries (or even unicode:chardata() which basically are io-lists but with unicode support) which is in many cases a better representation of strings. That means you can append two long strings with ["long string 1..", "long string 2.."] and the string will be "flattened" when output to a file or socket. So you can see the string module as an introduction how you should handle strings in erlang efficiently. :-) Though, that said, many of the new functions are slower then the functionality the are replacing, optimizations for the ASCII/Latin-1 case are being worked on. I changed the api rather drastically I know that. The reason was that in the old 'C' inspired API, you searched first and returned an index, then split the string on that index. Since handling unicode, grapheme clusters, binaries and deep lists of "characters" increases the cost of traversing the input string, I went with a new API which combines the two calls into one. e.g. find() or take() which searches the string and returns the result directly, to avoid the extra traversal. I'm sorry. I've brought up this issue before and got lots of push back. > > > > But every time I look up tried and true and long-used string functions to > find that they are deprecated and will be dropped in future Erlang releases > my blood pressure soars. Both my wife and my doctor tell me that at my age > this is a dangerous thing. > > > > I do understand the importance and necessity of Unicode. And applaud the > addition of Unicode functions. > > > > But the deprecated string functions have a long history. The English > language and Latin-1 characters are widely used around the world. > > > > Yes, it should be easy for programmers to translate code from one user > language to another. But I'm not convinced that the Gods of Erlang have > found the optimal solution by dropping all Latin-1 string functions. > > We have not said we will drop them, only deprecate them. They will stay with us for a long time. I want to remove them from the docs, because the manual page becomes a monster, but we will see what happens with that idea. > > > My particular application is directed toward English speakers. So, until > further notice, I have no use for Unicode. > > > > I don't want to sound like nationalist pig, but I think dropping the > Latin-1 string functions from future Erlang releases is a BIG mistake. > > > > I look up tokens/2, a function that I use fairly frequently, and I see > that it's deprecated. I look up the suggested replacement and I see > lexemes/2. > > > > So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I > see that a lexeme is "a meaningful linguistic unit." > > > > Meaning what? I just want to turn "this and that" into "This And That." > > So I had to google a lot to come up with a "tokens" replacement function name, a lexeme is what you think a token is or at least what I thought it was. see https://en.wikipedia.org/wiki/Lexical_analysis But 'tokens' is replaced by 'lexemes' and work exactly the same for ASCII lists as before with one exception [CR,LF] but more on that below. > > > I read further in the Erlang docs and I see "grapheme cluster." WHAT THE > ... IS GRAPHEME CLUSTER? > > This is best description that I found and what I wrote in the manual page: A *grapheme cluster* is a user-perceived character, which can be represented by several codepoints. I don't know if I can explain it better than that, that is the term used in the unicode standard, more information can be found there. Below that line I have the self explaining example: "?" [229] or [97, 778]"e?" [101, 778] So in Swedish we have the user-perceived character ? which is a 'a' with a dot above, that can be represented in unicode with codepoint 229 (?) or with the two codepoints 97 (a) 778 (dot above). So with that we can make "new" combined characters, as I tried with the 'e' and a dot above, which for me google chrome does not render correctly, the dot should be placed directly above the 'e'. This representation is important to avoid a "character" explosion for non LATIN-1 character sets such as in Asian and Arabic languages. You can change between the representations of '?' with unicode:characters_to_nf[c|d]_list. It is important that you normalize your data you get from the outside world to one representation before operating on it. But what they (the Unicode people) did was also define [CR,LF] as one grapheme which makes it impossible to use/extend the old functions in compatible way. So to split a multi-line string into lines, where you previously did: Lines = string:tokens(" a \n b \r\n c", "\r\n"), %% Split line on CR or LF You must now rewrite that to: Lines = string:lexemes(""a \n b \r\n c", ["\r\n",$n]), %% Split line on CR,LF or LF Blame the standard and not me :-) > > I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a > unit of a writing system." > > > > Ah yes, grapheme is defined in the docs. But I have to read and re-read > the definition to understand what the God's of Erlang mean by a "graphene > cluster." And I'm still not sure I get it. > > > > It sounds like someone took a linguistics class and is trying to show off. > > As you can see in this email my linguistic knowledge are way worse than yours, so maybe you can help and improve the manual. But it is tough to describe unicode handling in an easy way. When a character is not a character anymore it becomes fuzzy fast.. Everything here is from the top of my head and not tested, it's too late for that here and I have a Zelda boss to beat, my kids are way ahead of me. BR /Dan > > > But now I've spent 30 minutes--- time that I don't have to waste trying to > figure out how do a simple manipulation of "this and that." Recurse the > next time I want to look up a string function in the Erlang docs. > > > > SOLUTION > > > > Keep the Latin-1 string functions. Put them in a separate library if > necessary. Or put the new Unicode functions in a separate library. But > don't arbitrarily drop them. > > > > Some folks have suggested that I maintain my own library of the deprecated > Latin1 functions. But why should I have to do that? How does that help > other folks with the same issue? > > > > Bottom line: please please please do not drop the existing Latin-1 string > functions. > > > > Please don't. > > > > Best wishes, > > > > LRP > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Nov 22 22:55:32 2017 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 22 Nov 2017 16:55:32 -0500 (EST) Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <1511379790.16757707@apps.rackspace.com> <4402c210-5e88-b0ce-231f-1d3657fcdce8@gjunka.com> Message-ID: <1511387732.72033761@apps.rackspace.com> Hi All, Hardly a point in responses so far that I could disagree with or argue against. Although I still contend that the name "lexeme" is about as ugly a term as I could imagine. So a challenge: Can someone please write a short tutorial that shows on a one-to-one basis how to use Unicode functions to replace the Latin-1 functions then provide a link to it from the string docs. Or, better yet, simply integrate it into the string docs. Nevertheless, I'm still deeply troubled. Many thanks to all. Lloyd -----Original Message----- From: "Dan Gudmundsson" Sent: Wednesday, November 22, 2017 4:45pm To: "Grzegorz Junka" Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Strings - deprecated functions On Wed, Nov 22, 2017 at 8:59 PM Grzegorz Junka <[ list1@REDACTED ]( mailto:list1@REDACTED )> wrote: Dear Lloyd, Isn't this more about documentation than the code? What I am reading is that you want to keep the old functions because you don't understand how the new functions work. Shouldn't you rather ask for a more clear documentation? Is there anything in the old functions that is not supported in the new functions? GrzegorzJ On 22/11/2017 19:43, [ lloyd@REDACTED ]( mailto:lloyd@REDACTED ) wrote: Dear Gods of Erlang, "This module has been reworked in Erlang/OTP 20 to handle [ unicode:chardata() ]( http://erlang.org/doc/man/unicode.html#type-chardata ) and operate on grapheme clusters. The [ old functions ]( http://erlang.org/doc/man/string.html#oldapi ) that only work on Latin-1 lists as input are still available but should not be used. They will be deprecated in Erlang/OTP 21." The new functions also works on binaries (or even unicode:chardata() which basically are io-lists but with unicode support) which is in many cases a better representation of strings. That means you can append two long strings with ["long string 1..", "long string 2.."] and the string will be "flattened" when output to a file or socket. So you can see the string module as an introduction how you should handle strings in erlang efficiently. :-) Though, that said, many of the new functions are slower then the functionality the are replacing, optimizations for the ASCII/Latin-1 case are being worked on. I changed the api rather drastically I know that. The reason was that in the old 'C' inspired API, you searched first and returned an index, then split the string on that index. Since handling unicode, grapheme clusters, binaries and deep lists of "characters" increases the cost of traversing the input string, I went with a new API which combines the two calls into one. e.g. find() or take() which searches the string and returns the result directly, to avoid the extra traversal. I'm sorry. I've brought up this issue before and got lots of push back. But every time I look up tried and true and long-used string functions to find that they are deprecated and will be dropped in future Erlang releases my blood pressure soars. Both my wife and my doctor tell me that at my age this is a dangerous thing. I do understand the importance and necessity of Unicode. And applaud the addition of Unicode functions. But the deprecated string functions have a long history. The English language and Latin-1 characters are widely used around the world. Yes, it should be easy for programmers to translate code from one user language to another. But I'm not convinced that the Gods of Erlang have found the optimal solution by dropping all Latin-1 string functions. We have not said we will drop them, only deprecate them. They will stay with us for a long time. I want to remove them from the docs, because the manual page becomes a monster, but we will see what happens with that idea. My particular application is directed toward English speakers. So, until further notice, I have no use for Unicode. I don't want to sound like nationalist pig, but I think dropping the Latin-1 string functions from future Erlang releases is a BIG mistake. I look up tokens/2, a function that I use fairly frequently, and I see that it's deprecated. I look up the suggested replacement and I see lexemes/2. So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I see that a lexeme is "a meaningful linguistic unit." Meaning what? I just want to turn "this and that" into "This And That." So I had to google a lot to come up with a "tokens" replacement function name, a lexeme is what you think a token is or at least what I thought it was. see [ https://en.wikipedia.org/wiki/Lexical_analysis ]( https://en.wikipedia.org/wiki/Lexical_analysis ) But 'tokens' is replaced by 'lexemes' and work exactly the same for ASCII lists as before with one exception [CR,LF] but more on that below. I read further in the Erlang docs and I see "grapheme cluster." WHAT THE ... IS GRAPHEME CLUSTER? This is best description that I found and what I wrote in the manual page: A grapheme cluster is a user-perceived character, which can be represented by several codepoints. I don't know if I can explain it better than that, that is the term used in the unicode standard, more information can be found there. Below that line I have the self explaining example: "?" [229] or [97, 778]"e?" [101, 778] So in Swedish we have the user-perceived character ? which is a 'a' with a dot above, that can be represented in unicode with codepoint 229 (?) or with the two codepoints 97 (a) 778 (dot above). So with that we can make "new" combined characters, as I tried with the 'e' and a dot above, which for me google chrome does not render correctly, the dot should be placed directly above the 'e'. This representation is important to avoid a "character" explosion for non LATIN-1 character sets such as in Asian and Arabic languages. You can change between the representations of '?' with unicode:characters_to_nf[c|d]_list. It is important that you normalize your data you get from the outside world to one representation before operating on it. But what they (the Unicode people) did was also define [CR,LF] as one grapheme which makes it impossible to use/extend the old functions in compatible way. So to split a multi-line string into lines, where you previously did: Lines = string:tokens(" a \n b \r\n c", "\r\n"), %% Split line on CR or LF You must now rewrite that to: Lines = string:lexemes(""a \n b \r\n c", ["\r\n",$n]), %% Split line on CR,LF or LF Blame the standard and not me :-) I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a unit of a writing system." Ah yes, grapheme is defined in the docs. But I have to read and re-read the definition to understand what the God's of Erlang mean by a "graphene cluster." And I'm still not sure I get it. It sounds like someone took a linguistics class and is trying to show off. As you can see in this email my linguistic knowledge are way worse than yours, so maybe you can help and improve the manual. But it is tough to describe unicode handling in an easy way. When a character is not a character anymore it becomes fuzzy fast.. Everything here is from the top of my head and not tested, it's too late for that here and I have a Zelda boss to beat, my kids are way ahead of me. BR /Dan But now I've spent 30 minutes--- time that I don't have to waste trying to figure out how do a simple manipulation of "this and that." Recurse the next time I want to look up a string function in the Erlang docs. SOLUTION Keep the Latin-1 string functions. Put them in a separate library if necessary. Or put the new Unicode functions in a separate library. But don't arbitrarily drop them. Some folks have suggested that I maintain my own library of the deprecated Latin1 functions. But why should I have to do that? How does that help other folks with the same issue? Bottom line: please please please do not drop the existing Latin-1 string functions. Please don't. Best wishes, LRP _______________________________________________erlang-questions mailing list[ erlang-questions@REDACTED ]( mailto:erlang-questions@REDACTED )[ http://erlang.org/mailman/listinfo/erlang-questions ]( http://erlang.org/mailman/listinfo/erlang-questions )_______________________________________________ erlang-questions mailing list [ erlang-questions@REDACTED ]( mailto:erlang-questions@REDACTED ) [ http://erlang.org/mailman/listinfo/erlang-questions ]( http://erlang.org/mailman/listinfo/erlang-questions ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Wed Nov 22 23:22:07 2017 From: t@REDACTED (Tristan Sloughter) Date: Wed, 22 Nov 2017 14:22:07 -0800 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511387732.72033761@apps.rackspace.com> References: <1511379790.16757707@apps.rackspace.com> <4402c210-5e88-b0ce-231f-1d3657fcdce8@gjunka.com> <1511387732.72033761@apps.rackspace.com> Message-ID: <1511389327.238462.1181544704.7766B29E@webmail.messagingengine.com> > Can someone please write a short tutorial that shows on a one-to-one > basis how to use Unicode functions to replace the Latin-1 functions > then provide a link to it from the string docs. Or, better yet, simply > integrate it into the string docs. In your first email you already mentioned the fact that the docs do provide a one-to-one mapping, it links directly to the new function to use from the doc of the deprecated function. Like in your case of tokens to lexemes the same arguments work and the functions have examples, so what is missing from the docs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Nov 22 23:50:58 2017 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 22 Nov 2017 17:50:58 -0500 (EST) Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511389327.238462.1181544704.7766B29E@webmail.messagingengine.com> References: <1511379790.16757707@apps.rackspace.com> <4402c210-5e88-b0ce-231f-1d3657fcdce8@gjunka.com> <1511387732.72033761@apps.rackspace.com> <1511389327.238462.1181544704.7766B29E@webmail.messagingengine.com> Message-ID: <1511391058.400425931@apps.rackspace.com> Hi Tristan, When I have a few moments of free time, I'll go through the string docs carefully and try to point out all the issues that I find confusing or obscure. Maybe I'm the only person in the world with issues here. If so, I'll shut up and try to get with the program. But I can't tell you how much time I've wasted trying to grasp how and when to use various functions in the docs--- and certainly not only the string documentation. I've been trying for more than three years now to master Erlang sufficiently well to build my current project. The canonical books have made it possible (thanks, Joe, et. al.) But I'm an army of one. I can't turn to the programmer next me or my programming supervisor to ask what's what. This list is a god-send and the folks on it are extraordinarily gracious and generous. But for all that, there are still major libraries and functions that I've looked at thinking, hey, that might be useful. But I can't figure out for the life me how and when to use them even though they might be quite useful. If we see reason to leave these obscure functions in, why can't we leave the Latin-1 functions? Even if only in name as a wrapper around the Unicode functions that deliver the same functionality? When I launch my current project, I'll be happy to dig in deep and do what I can to help improve the docs. The best that I can offer is my profound ignorance and willingness to ask Micky the Dunce questions. One Erlanger pointed out that they would rather have maintainers work on bug fixes and new features than documentation. I'd argue that without clear and inviting documentation we discourage adopters and the cripple the vitality of our community. I'm eager to do what I can do improve the documentation. But I can't spend much time at it until I get paid. And I don't get paid until I've launched my current project. All the best, Lloyd -----Original Message----- From: "Tristan Sloughter" Sent: Wednesday, November 22, 2017 5:22pm To: erlang-questions@REDACTED Subject: Re: [erlang-questions] Strings - deprecated functions Can someone please write a short tutorial that shows on a one-to-one basis how to use Unicode functions to replace the Latin-1 functions then provide a link to it from the string docs. Or, better yet, simply integrate it into the string docs. In your first email you already mentioned the fact that the docs do provide a one-to-one mapping, it links directly to the new function to use from the doc of the deprecated function. Like in your case of tokens to lexemes the same arguments work and the functions have examples, so what is missing from the docs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Nov 23 03:36:35 2017 From: ok@REDACTED (ok@REDACTED) Date: Thu, 23 Nov 2017 15:36:35 +1300 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511379790.16757707@apps.rackspace.com> References: <1511379790.16757707@apps.rackspace.com> Message-ID: <4d65e8b2d882c2bcce5cb1f459165958.squirrel@chasm.otago.ac.nz> lloyd@REDACTED wrote: > I read further in the Erlang docs and I see "grapheme cluster." WHAT THE > ... IS GRAPHEME CLUSTER? > > I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a > unit of a writing system." "Grapheme" and "grapheme cluster" are technical terms in Unicode. The best place to look is probably UAX#29 Unicode Text Segmentantion http://unicode.org/reports/tr29/ Section 3 begins with this paragraph, which should help: It is important to recognize that what the user thinks of as a ?character??a basic unit of a writing system for a language?may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, ?G? + acute-accent is a user-perceived character: users think of it as a single character, yet is actually represented by two Unicode code points. These user-perceived characters are approximated by what is called a grapheme cluster, which can be determined programmatically. > It sounds like someone took a linguistics class and is trying to show off. It would be pretty horrifying if many of the people defining Unicode hadn't taken a linguistics class or three... It's actually a very obvious practical problem: suppose you are in your favorite editor and press the "move forward 1 character" key. The distinction between Unicode and UTF-8 makes it sufficiently clear that this doesn't mean "move forward one byte" (C), and the distinction between Unicode and UTF-16 makes it sufficiently clear that it doesn't mean "move forward one 16-bit char" (Java). But it doesn't mean "move forward one Unicode code point" either. There is no limit in principle to the number of code points in a user-perceived character. Figuring out just how many code points in a "character" (= grapheme cluster) is sufficiently tricky that you do not want to do it yourself. The text *I* generate is almost exclusively Latin-1, but it is less and less common for me to *get* data in that form. I too would like full retention of Latin-1 support >>for data I am fully in control of<<. From mononcqc@REDACTED Thu Nov 23 03:45:55 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 22 Nov 2017 21:45:55 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511379790.16757707@apps.rackspace.com> References: <1511379790.16757707@apps.rackspace.com> Message-ID: <20171123024552.GD2024@ferdmbp.local> On 11/22, lloyd@REDACTED wrote: >I read further in the Erlang docs and I see "grapheme cluster." WHAT >THE ... IS GRAPHEME CLUSTER? > A quick run-through. In ASCII and latin-1 you mostly can deal with the following words, which are all synonymous: - character - letter - symbol In some variants, you also have to add the word 'diacritic' or 'accent' which let you modify a character in terms of linguistincs: a + ` = ? Fortunately, in latin1, most of these naughty diacritics have been bundled into specific characters. In French, for example, this final 'letter' can be represented under a single code (224). There are however complications coming from that. One of them is 'collation' (the sorting order of letters). For example, a and ? in French ought to sort in the same portion of the alphabet (before 'b'), but by default, they end up sorting after 'z'. In Danish, A is the first letter of the alphabet, but ? is last. Also ? is seen as a ligature of Aa; Aa is sorted like ? rather than two letters 'Aa' one after the other. Swedish has different diacritics with different orders: ?, ?, ?. So uh, currently, Erlang did not even do a great job at Latin-1 because there was nothing to handle 'collations' (string comparisons to know what is equal or not). Enter UNICODE. To make a matter short (and I hope Mr. ROK won't be too mad at my gross oversimplifications), we have the following terms in vocabulary: - character: smallest representable unit in a language, in the abstract. '`' is a character, so is 'a', and so is '?' - glyph: the visual representation of a character. Think of it as a character from the point of view of the font or typeface designer. For example, the same glyph may be used for the capital letter 'pi' and the mathematical symbol for a product: ?. Similarly, capital 'Sigma' and the mathematical 'sum' may have different character representation, but the same ? glyph. - letter: an element of an alphabet - codepoint: A given value in the unicode space. There's a big table with a crapload of characters in them, and every character is assigned a codepoint, as a unique identifier for it. - code unit: a specific encoding of a given code point. This refers to bits, not just the big table. The same code point may have different code units in UTF-8, UTF-16, and UTF-32, which are 3 'encodings' of unicode. - grapheme: what the user thinks of a 'character' - grapheme cluster: what you want to think of as a 'character' for your user's sake. Basically, 'a' and '`' can be two graphemes, but if I combine them together as '?', I want to be able to say that a single 'delete' key press will remove both the '`' and the 'a' at once from my text, and not be left with one or the other. We're left with the word 'lexeme' which is not really defined in the unicode glossary. Linguists will treat it as a lexical unit (word or term of vocabulary). In computer talk, you'd just define it as an arbitrary string, or maybe token (it appears some people use them interchangeably). The big fun bit is that unicode takes all these really shitty complicated linguistic things and specifies how they should be handled. Like, what makes two strings equal? I understand it's of little importance in English, but the french '?' can be represented both as a single ? or as e+?. It would be good, when you deal with say JSON or maybe my username, that you don't end up having 'Fr?d?ric' as 4 different people depending on which form was used. JSON, by the way, specifies 'unicode' as an encoding! In any case, these encoding rules are specified in normalization forms (http://unicode.org/reports/tr15/). The new interface lets you compare string with 'string:compare(A, B, IgnoreCase, nfc | nfk | nfkc | nfkd)' which is good, because the rules for changing case are also language- or alphabet-specific. So when you look at functions like 'string:next_grapheme/1' and 'string:next_codepoint/1', they're related to whether you want to consume the data in terms of user-observable 'characters' or in terms of unicode-specific 'characters'. Because they're not the same, and depending on what you want to do, this is important. You could call 'string:to_graphemes' and get an iterable list the way you could use them before: 1> string:to_graphemes("??e?"). [223,8593,[101,778]] 2> string:to_graphemes(<<"??e?"/utf8>>). [223,8593,[101,778]] But now it's working regardless of the initial format! This is really freaking cool. >SOLUTION > Translation! centre/2-3 ==> pad/2-4 Same thing, except pad is more generic and accepts a direction chars/2 ==> lists:duplicate/2 Same thing, except the 2 arguments are flipped. chars/3 ==> ??? No direct match, but just call [lists:duplicate(N, Elem)|Tail] to get an equivalence chr/2 ==> find/2-3 (with 3rd argument 'leading') whereas chr/2 returns a position, find/2-3 returns the string after the match. This leaves a bit of a gap if you're looking to take everyting *until* a given character (look at take/3-4 if you need a single character, or maybe string:split/2-3), or really the position, but in Unicode the concept of a position is vague: is it based on code units, codepoints, grapheme clusters, or what? concat/2 ==> ??? You can concatenate strings by using iolists: [A,B]. If you need to flatten the string with unicode:character_to_[list|binary]. copies/2 ==> lists:duplicate/2 Same thing, except the two arguments are flipped cspan/2 ==> take/3-4 specifically, cspan(Str, Chars) is equivalent to take(Str, Chars, false, leading). Returns a pair of {Before, After} strings rather than a length. join/2 ==> lists:join/2 same thing, but the arguments are flipped left/2-3 ==> pad/2-4 same thing, except pad is more generic and accepts a direction len/1 ==> length/1 returns grapheme cluster counts rather than 'characters'. rchr/2 ==> find/2-3 (with 3rd argument 'trailing') see chr/2 conversion for description. right/2-3 ==> pad/2-4 same as center/2-3 and left/2-3. rstr/2 ==> find/3 use 'trailing' as third argument for similar semantics. Drops characters before the match and returns the leftover string rather than just an index. A bit of a gap if you want the opposite, maybe use string:split/2-3 span/2 ==> take/2 no modifications required for arguments, but take/2 returns a {Before, After} pair of strings rather than a length. str/2 ==> find/2 use 'leading' as a third argument. Drops characters before the match rather than just an index. Maybe string:split/2-3 is nicer there? strip/1-3 ==> trim/1-3 Same, aside from the direction. strip/2-3 accepted 'left | right | both' whereas trim/2-3 accepts 'leading | trailing | both'. Be careful. Oh also strip/3 takes a single character as an argument and trim/3 takes a list of characters. sub_string/2-3 ==> slice/2-3 (String, Start, Stop) is changed for (String, Start, Length). This reflects the idea that grapheme clusters make it a lot harder to know where in a string a specific position is. The length is in grapheme clusters. substr/2-3 ==> slice/2-3 no change sub_word/2-3 ==> nth_lexeme/3 Same, except rather than a single character in the last position, it now takes a list of separators (grapheme clusters). So ".e" is actually the list of [$., $e], two distinct separators. to_lower/1 ==> lowercase/1 Same to_upper/1 ==> uppercase/1 Same tokens/2 ==> lexemes/2 Same words/2 ==> lexemes/2 Same, but lexemes/2 accepts a list of 'characters' (grapheme clusters) instead of a single one of them. The biggest annoyance I have had converting so far was handling find/2-3; in a lot of places in code, I had patterns where the objective was to drop what happend *after* a given character, and the function does just the opposite. You can take a look at string:split/2-3 there. The second biggest annoyance is making sure that functions that used to take just a single character now may take more than one of them. It makes compatibility a bit weird. >Keep the Latin-1 string functions. Put them in a separate library if >necessary. Or put the new Unicode functions in a separate library. But >don't arbitrarily drop them. > >Some folks have suggested that I maintain my own library of the >deprecated Latin1 functions. But why should I have to do that? How does >that help other folks with the same issue? > I had a few problems with it myself; I just finished updating rebar3 and dependencies to run on both >OTP-21 releases and stuff dating back to OTP-16. The problem we have is that we run on a backwards compat schedule that is stricter and longer than the OTP team. For example, string:join/2 is being replaced with lists:join/2, but lists:join/2 did not exist in R16 and string:join/2 is deprecated in OTP-21. So we needed to extract that function from OTP into custom modules everywhere, and replace old usage with new one. I was forced to add translation functions and modules like https://github.com/erlang/rebar3/blob/master/src/rebar_string.erl to the code base, along with this conditional define: https://github.com/erlang/rebar3/blob/master/rebar.config#L33 It's a bit painful, but it ends up working quite alright. Frankly it's nicer if you can adopt OTP's deprecation pace, it's quite painful for us being on a larger sequence. >Bottom line: please please please do not drop the existing Latin-1 >string functions. > >Please don't. > >Best wishes, > >LRP > It's probably alright if they keep warnings for a good period of time before dropping everything. OTP-21 starts warning so people who want to keep 'warning_as_errors' as an option will suffer the most. But overall, you can't escape Unicode. Work with JSON? There it is. HTTP? There again. URLs? You bet. File systems? Hell yes! Erlang modules and app files: also yes! The one place I've seen a deliberate attempt at not doing it was with host names in certificate validation (or DNS), since there, making distinct domain names compare the same could be an attack vector. There you get to deal with the magic of punycode (https://en.wikipedia.org/wiki/Punycode) if you want to be safer. - Fred. From lloyd@REDACTED Thu Nov 23 04:54:19 2017 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Wed, 22 Nov 2017 22:54:19 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <20171123024552.GD2024@ferdmbp.local> References: <1511379790.16757707@apps.rackspace.com> <20171123024552.GD2024@ferdmbp.local> Message-ID: <7D1ED654-8470-4717-9015-ADA93CAD01D4@writersglen.com> A big hearty thanks to ok and Fred for the terrific clarifications. I guess I'll just have to suck it up and convert all Latin-1 functions that I've written so far to Unicode functions. If I wait until later I may not be among the living. Hate to foist that off on some unsuspecting soul. Meanwhile, I've just pushed my release target out another month (or two). And thanks to all. Lloyd Sent from my iPad > On Nov 22, 2017, at 9:45 PM, Fred Hebert wrote: > >> On 11/22, lloyd@REDACTED wrote: >> I read further in the Erlang docs and I see "grapheme cluster." WHAT THE ... IS GRAPHEME CLUSTER? >> > > A quick run-through. In ASCII and latin-1 you mostly can deal with the following words, which are all synonymous: > > - character > - letter > - symbol > > In some variants, you also have to add the word 'diacritic' or 'accent' which let you modify a character in terms of linguistincs: > > a + ` = ? > > Fortunately, in latin1, most of these naughty diacritics have been bundled into specific characters. In French, for example, this final 'letter' can be represented under a single code (224). > > There are however complications coming from that. One of them is 'collation' (the sorting order of letters). For example, a and ? in French ought to sort in the same portion of the alphabet (before 'b'), but by default, they end up sorting after 'z'. > > In Danish, A is the first letter of the alphabet, but ? is last. Also ? is seen as a ligature of Aa; Aa is sorted like ? rather than two letters 'Aa' one after the other. Swedish has different diacritics with different orders: ?, ?, ?. > > So uh, currently, Erlang did not even do a great job at Latin-1 because there was nothing to handle 'collations' (string comparisons to know what is equal or not). > > > Enter UNICODE. To make a matter short (and I hope Mr. ROK won't be too mad at my gross oversimplifications), we have the following terms in vocabulary: > > - character: smallest representable unit in a language, in the abstract. '`' is a character, so is 'a', and so is '?' > - glyph: the visual representation of a character. Think of it as a character from the point of view of the font or typeface designer. For example, the same glyph may be used for the capital letter 'pi' and the mathematical symbol for a product: ?. Similarly, capital 'Sigma' and the mathematical 'sum' may have different character representation, but the same ? glyph. > - letter: an element of an alphabet > - codepoint: A given value in the unicode space. There's a big table with a crapload of characters in them, and every character is assigned a codepoint, as a unique identifier for it. > - code unit: a specific encoding of a given code point. This refers to bits, not just the big table. The same code point may have different code units in UTF-8, UTF-16, and UTF-32, which are 3 'encodings' of unicode. > - grapheme: what the user thinks of a 'character' > - grapheme cluster: what you want to think of as a 'character' for your user's sake. Basically, 'a' and '`' can be two graphemes, but if I combine them together as '?', I want to be able to say that a single 'delete' key press will remove both the '`' and the 'a' at once from my text, and not be left with one or the other. > > We're left with the word 'lexeme' which is not really defined in the unicode glossary. Linguists will treat it as a lexical unit (word or term of vocabulary). In computer talk, you'd just define it as an arbitrary string, or maybe token (it appears some people use them interchangeably). > > The big fun bit is that unicode takes all these really shitty complicated linguistic things and specifies how they should be handled. > > Like, what makes two strings equal? I understand it's of little importance in English, but the french '?' can be represented both as a single ? or as e+?. It would be good, when you deal with say JSON or maybe my username, that you don't end up having 'Fr?d?ric' as 4 different people depending on which form was used. JSON, by the way, specifies 'unicode' as an encoding! > > In any case, these encoding rules are specified in normalization forms (http://unicode.org/reports/tr15/). The new interface lets you compare string with 'string:compare(A, B, IgnoreCase, nfc | nfk | nfkc | nfkd)' which is good, because the rules for changing case are also language- or alphabet-specific. > > So when you look at functions like 'string:next_grapheme/1' and 'string:next_codepoint/1', they're related to whether you want to consume the data in terms of user-observable 'characters' or in terms of unicode-specific 'characters'. Because they're not the same, and depending on what you want to do, this is important. > > You could call 'string:to_graphemes' and get an iterable list the way you could use them before: > > 1> string:to_graphemes("??e?"). > [223,8593,[101,778]] > 2> string:to_graphemes(<<"??e?"/utf8>>). > [223,8593,[101,778]] > > But now it's working regardless of the initial format! This is really freaking cool. > >> SOLUTION >> > > Translation! > > centre/2-3 ==> pad/2-4 > Same thing, except pad is more generic and accepts a direction > chars/2 ==> lists:duplicate/2 > Same thing, except the 2 arguments are flipped. > chars/3 ==> ??? > No direct match, but just call [lists:duplicate(N, Elem)|Tail] to get an equivalence > chr/2 ==> find/2-3 (with 3rd argument 'leading') > whereas chr/2 returns a position, find/2-3 returns the string after the match. This leaves a bit of a gap if you're looking to take everyting *until* a given character (look at take/3-4 if you need a single character, or maybe string:split/2-3), or really the position, but in Unicode the concept of a position is vague: is it based on code units, codepoints, grapheme clusters, or what? > concat/2 ==> ??? > You can concatenate strings by using iolists: [A,B]. If you need to flatten the string with unicode:character_to_[list|binary]. > copies/2 ==> lists:duplicate/2 > Same thing, except the two arguments are flipped > cspan/2 ==> take/3-4 > specifically, cspan(Str, Chars) is equivalent to take(Str, Chars, false, leading). Returns a pair of {Before, After} strings rather than a length. > join/2 ==> lists:join/2 > same thing, but the arguments are flipped > left/2-3 ==> pad/2-4 > same thing, except pad is more generic and accepts a direction > len/1 ==> length/1 > returns grapheme cluster counts rather than 'characters'. > rchr/2 ==> find/2-3 (with 3rd argument 'trailing') > see chr/2 conversion for description. > right/2-3 ==> pad/2-4 > same as center/2-3 and left/2-3. > rstr/2 ==> find/3 > use 'trailing' as third argument for similar semantics. Drops characters before the match and returns the leftover string rather than just an index. A bit of a gap if you want the opposite, maybe use string:split/2-3 > span/2 ==> take/2 > no modifications required for arguments, but take/2 returns a {Before, After} pair of strings rather than a length. > str/2 ==> find/2 > use 'leading' as a third argument. Drops characters before the match rather than just an index. Maybe string:split/2-3 is nicer there? > strip/1-3 ==> trim/1-3 > Same, aside from the direction. strip/2-3 accepted 'left | right | both' whereas trim/2-3 accepts 'leading | trailing | both'. Be careful. Oh also strip/3 takes a single character as an argument and trim/3 takes a list of characters. > sub_string/2-3 ==> slice/2-3 > (String, Start, Stop) is changed for (String, Start, Length). This reflects the idea that grapheme clusters make it a lot harder to know where in a string a specific position is. The length is in grapheme clusters. > substr/2-3 ==> slice/2-3 > no change > sub_word/2-3 ==> nth_lexeme/3 > Same, except rather than a single character in the last position, it now takes a list of separators (grapheme clusters). So ".e" is actually the list of [$., $e], two distinct separators. > to_lower/1 ==> lowercase/1 > Same > to_upper/1 ==> uppercase/1 > Same > tokens/2 ==> lexemes/2 > Same > words/2 ==> lexemes/2 > Same, but lexemes/2 accepts a list of 'characters' (grapheme clusters) instead of a single one of them. > > > The biggest annoyance I have had converting so far was handling find/2-3; in a lot of places in code, I had patterns where the objective was to drop what happend *after* a given character, and the function does just the opposite. You can take a look at string:split/2-3 there. > > The second biggest annoyance is making sure that functions that used to take just a single character now may take more than one of them. It makes compatibility a bit weird. > >> Keep the Latin-1 string functions. Put them in a separate library if necessary. Or put the new Unicode functions in a separate library. But don't arbitrarily drop them. >> >> Some folks have suggested that I maintain my own library of the deprecated Latin1 functions. But why should I have to do that? How does that help other folks with the same issue? >> > > I had a few problems with it myself; I just finished updating rebar3 and dependencies to run on both >OTP-21 releases and stuff dating back to OTP-16. The problem we have is that we run on a backwards compat schedule that is stricter and longer than the OTP team. > > For example, string:join/2 is being replaced with lists:join/2, but lists:join/2 did not exist in R16 and string:join/2 is deprecated in OTP-21. So we needed to extract that function from OTP into custom modules everywhere, and replace old usage with new one. > > I was forced to add translation functions and modules like https://github.com/erlang/rebar3/blob/master/src/rebar_string.erl to the code base, along with this conditional define: https://github.com/erlang/rebar3/blob/master/rebar.config#L33 > > It's a bit painful, but it ends up working quite alright. Frankly it's nicer if you can adopt OTP's deprecation pace, it's quite painful for us being on a larger sequence. > >> Bottom line: please please please do not drop the existing Latin-1 string functions. >> >> Please don't. >> >> Best wishes, >> >> LRP >> > > It's probably alright if they keep warnings for a good period of time before dropping everything. OTP-21 starts warning so people who want to keep 'warning_as_errors' as an option will suffer the most. > > But overall, you can't escape Unicode. Work with JSON? There it is. HTTP? There again. URLs? You bet. File systems? Hell yes! Erlang modules and app files: also yes! > > The one place I've seen a deliberate attempt at not doing it was with host names in certificate validation (or DNS), since there, making distinct domain names compare the same could be an attack vector. There you get to deal with the magic of punycode (https://en.wikipedia.org/wiki/Punycode) if you want to be safer. > > - Fred. From lloyd@REDACTED Thu Nov 23 05:18:04 2017 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Wed, 22 Nov 2017 23:18:04 -0500 Subject: [erlang-questions] Need help Message-ID: <39AB3939-0AD3-49FC-B3DD-7CCB66A3F7A6@writersglen.com> Hello, I'm not sure that this is the right place to ask, but I'm building a creative space on the web for writers, readers and publishing professionals. To hit my release target I need help with a few short-term sequential Erlang tasks. Most tasks, I'd venture, can be completed by an Erlang wizard in a day or two or less. I'm self-funded, but, naturally, I can pay. If interested, zip me an email with availability, experience, and pay requirements. Thanks, Lloyd Sent from my iPad From arunp@REDACTED Thu Nov 23 07:36:18 2017 From: arunp@REDACTED (Arun) Date: Thu, 23 Nov 2017 12:06:18 +0530 Subject: [erlang-questions] Mnesia deleting log file In-Reply-To: References: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> Message-ID: Even after running the application for 2 days I do a power off of the device, I'm facing this problem. Currently, the value of "dump_log_write_threshold" configuration is set to 1. Anyhow, I'll try to reproduce the problem by setting the "dump_log_update_in_place" to false and intimate the status. On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote: > > I agree?setting dump_log_in_place to false makes extremely unlikely > for mnesia to fail in this way. Both JFFS2 and UBIFS ensure that > rename is atomic, even when power is pulled. > > For desktop applications, the culprit is likely the VM being killed, > not the whole system crashing. Thus the rename was either executed or > not; either way we have a complete DAT file. > > There is always a chance of things like this happening: > http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but > this shouldn?t happen on most hardware. > > Note that the default for dump_log_time_threshold is 3 minutes. Speed > vs Safety is always a trade-off, but many people will be surprised to > lose data in a crash 2 minutes after saving. > > *From:*Jesper Louis Andersen [mailto:jesper.louis.andersen@REDACTED] > *Sent:* Wednesday, November 22, 2017 7:54 AM > *To:* Michael Schmidt > *Cc:* Arun ; erlang-questions@REDACTED > *Subject:* Re: [erlang-questions] Mnesia deleting log file > > This thread makes we wonder about the following section in the Mnesia > documentation: > > "If a power failure occurs during the dump, this can cause the > randomly accessed DAT files to become corrupt. If the parameter is set > to false, Mnesia copies the DAT files and target the dump to the new > temporary files. If the dump is successful, the temporary files are > renamed to their normal DAT suffixes. The possibility for > unrecoverable inconsistencies in the data files becomes much smaller > with this strategy. However, the actual dumping of the transaction log > becomes considerably slower. The system designer must decide whether > speed or safety is the higher priority." > > In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an > OS crash might leave you with a temporary file and the target file, > hardlinked as if ln() was called, but the move itself is atomic, even > under OS crashes. Of course, this assumes you correctly wrote your > temporary file, fsync()'ed the file and then performed the rename() as > you should. > > With that information in mind, what is the scenario which can make > mnesia fail if the dump_log_update_in_place is set to false? There may > be a window, but the operating system cannot be the provider of this > window at all, unless it is incorrectly implemented or configured[1]. > Hence my question. > > [1] The primary concern are write caches on disks which can have a > write hole unless they have proper (working!) battery backup of their > cache or are able to write their memory to a small solid state part of > the chip. Or if you are using ZFS where copy-on-write semantics > completely eliminates the write hole. > > On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt > > wrote: > > Hi Arun, > > If you happen to reboot when Mnesia is writing out its logfile, it > can become corrupt (and lose data). We fight this issue on > embedded devices a lot. > > Look ?Configuration Parameters? section here: > > http://erlang.org/doc/man/mnesia.html#id69584 > > You will want to set dump_log_update_in_placeto false. This will > make it write out the table to a new file, and rename it the very > end. > > One other option to look at is dump_log_time_thresholdto change > how often items are flushed to disk. For configuration data, you > can likely set this to a few seconds. > > These options can be set via the sys.config file as well > > Mike > > *From:*erlang-questions-bounces@REDACTED > [mailto:erlang-questions-bounces@REDACTED > ] *On Behalf Of *Arun > *Sent:* Tuesday, November 21, 2017 10:16 PM > *To:* erlang-questions@REDACTED > *Subject:* [erlang-questions] Mnesia deleting log file > > Dear all, > > I've a program written in erlang which uses Mnesia application as > the database application. I've a table created by name > "configuration_table" which stores > > certain configurations that need to be persistent. Occasionally, > whenever I restart my program the following error is thrown by > mnesia and I end up losing > > all the persistent configurations. > > I've searched about this problem in the erlang documentation and > all it tells is "Node not running". What could be the probable > cause for this problem > > and how do I fix it? > > Mnesia('test@REDACTED '): Data may > be missing, Corrupt logfile deleted: > "/home/utl/mnesia_database/configuration_table.DCL", > {node_not_running, > 'test@REDACTED '} > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud > service. > ______________________________________________________________________ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > ______________________________________________________________________ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Thu Nov 23 08:49:31 2017 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Thu, 23 Nov 2017 08:49:31 +0100 Subject: [erlang-questions] Worth minimizing the number of inter-node messages? Message-ID: Hi all enlightened people, Suppose that you, in a distributed Erlang setup, have a process A, running on one of the nodes, which, in its state, holds pids B1, B2, ..., Bn, where each process B could be running on any of the nodes in the cluster. Suppose further that you want to typically broadcast messages to all processes (B1 through Bn) - the same message to all processes (you can think of it as a "group chat" if you like where "A" is the room and B1-Bn are the users joining the room - seeing everything posted there - or you could see it as a message bus for a topic "A"). The number n here could be in the order of "2-10 thousands" and messages are typically not huge but perhaps not tiny either (< 1kB). In such a setup, if you want to minimize the time to distribute the message to all processes B1 to Bn (the time between when the first B receives the message to the last), would you consider it be worth to try to minimize the amount of traffic between the nodes, by, let's say, grouping all processes belonging to the same node and instead adding such groups to the process A? Then there would be one message going between the nodes and then the local groups on each node take care of sending it to all its local processes (under the assumption that local message passing has a lower cost than inter-node message passing). Is there even such library out there that does something similar? I know of pg and pg2 but the goals of these modules is not about minimizing node-to-node traffic if I understand them correctly. Or perhaps others are approaching such broadcasting of messages in different ways that does not rely on Erlang message passing at all - such as relying on multicast or using message buses (such as Rabbit or NATS)? Perhaps that is a better approach? I suppose "measuring" is the answer here...but perhaps someone has done similar comparisons already? /Stefan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Thu Nov 23 08:55:27 2017 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Thu, 23 Nov 2017 08:55:27 +0100 Subject: [erlang-questions] Worth minimizing the number of inter-node messages? In-Reply-To: References: Message-ID: <4C760A9B-FD45-48F1-BA17-A6E1BAA42E88@gmail.com> You neglected to mention a critical aspect. How often are you broadcasting? Regards, Sergej > On 23 Nov 2017, at 08:49, Stefan Hellkvist wrote: > > Hi all enlightened people, > > Suppose that you, in a distributed Erlang setup, have a process A, running on one of the nodes, which, in its state, holds pids B1, B2, ..., Bn, where each process B could be running on any of the nodes in the cluster. Suppose further that you want to typically broadcast messages to all processes (B1 through Bn) - the same message to all processes (you can think of it as a "group chat" if you like where "A" is the room and B1-Bn are the users joining the room - seeing everything posted there - or you could see it as a message bus for a topic "A"). The number n here could be in the order of "2-10 thousands" and messages are typically not huge but perhaps not tiny either (< 1kB). > > In such a setup, if you want to minimize the time to distribute the message to all processes B1 to Bn (the time between when the first B receives the message to the last), would you consider it be worth to try to minimize the amount of traffic between the nodes, by, let's say, grouping all processes belonging to the same node and instead adding such groups to the process A? Then there would be one message going between the nodes and then the local groups on each node take care of sending it to all its local processes (under the assumption that local message passing has a lower cost than inter-node message passing). Is there even such library out there that does something similar? I know of pg and pg2 but the goals of these modules is not about minimizing node-to-node traffic if I understand them correctly. > > Or perhaps others are approaching such broadcasting of messages in different ways that does not rely on Erlang message passing at all - such as relying on multicast or using message buses (such as Rabbit or NATS)? Perhaps that is a better approach? I suppose "measuring" is the answer here...but perhaps someone has done similar comparisons already? > > /Stefan > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From hellkvist@REDACTED Thu Nov 23 09:03:51 2017 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Thu, 23 Nov 2017 09:03:51 +0100 Subject: [erlang-questions] Worth minimizing the number of inter-node messages? In-Reply-To: <4C760A9B-FD45-48F1-BA17-A6E1BAA42E88@gmail.com> References: <4C760A9B-FD45-48F1-BA17-A6E1BAA42E88@gmail.com> Message-ID: Each process (or "user" if you prefer the group chat analogy) could send a message, say once every second. So if there are 1000 processes in one such "room" there are 1000 messages per second. Then there could be many such "rooms" also in the system, running on different nodes, but that is in the order of 10 at most. On Thu, Nov 23, 2017 at 8:55 AM, Sergej Jure?ko wrote: > You neglected to mention a critical aspect. How often are you broadcasting? > > Regards, > Sergej > > > On 23 Nov 2017, at 08:49, Stefan Hellkvist wrote: > > > > Hi all enlightened people, > > > > Suppose that you, in a distributed Erlang setup, have a process A, > running on one of the nodes, which, in its state, holds pids B1, B2, ..., > Bn, where each process B could be running on any of the nodes in the > cluster. Suppose further that you want to typically broadcast messages to > all processes (B1 through Bn) - the same message to all processes (you can > think of it as a "group chat" if you like where "A" is the room and B1-Bn > are the users joining the room - seeing everything posted there - or you > could see it as a message bus for a topic "A"). The number n here could be > in the order of "2-10 thousands" and messages are typically not huge but > perhaps not tiny either (< 1kB). > > > > In such a setup, if you want to minimize the time to distribute the > message to all processes B1 to Bn (the time between when the first B > receives the message to the last), would you consider it be worth to try to > minimize the amount of traffic between the nodes, by, let's say, grouping > all processes belonging to the same node and instead adding such groups to > the process A? Then there would be one message going between the nodes and > then the local groups on each node take care of sending it to all its local > processes (under the assumption that local message passing has a lower cost > than inter-node message passing). Is there even such library out there that > does something similar? I know of pg and pg2 but the goals of these modules > is not about minimizing node-to-node traffic if I understand them correctly. > > > > Or perhaps others are approaching such broadcasting of messages in > different ways that does not rely on Erlang message passing at all - such > as relying on multicast or using message buses (such as Rabbit or NATS)? > Perhaps that is a better approach? I suppose "measuring" is the answer > here...but perhaps someone has done similar comparisons already? > > > > /Stefan > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tobias.Schlager@REDACTED Thu Nov 23 09:13:54 2017 From: Tobias.Schlager@REDACTED (Tobias Schlager) Date: Thu, 23 Nov 2017 08:13:54 +0000 Subject: [erlang-questions] Worth minimizing the number of inter-node messages? In-Reply-To: References: Message-ID: <12F2115FD1CCEE4294943B2608A18FA301C2A10269@MAIL01.win.lbaum.eu> Hi Stefan, your question reminds me of a Discord blog about inter-node messaging I read, some time ago. I was lucky enough to be able to find it again. I know, it's about Elixir, but the concepts apply nonetheless. Here you go [1]. Regards Tobias [1] https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Nov 23 09:17:34 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 23 Nov 2017 09:17:34 +0100 Subject: [erlang-questions] map elements in defined order In-Reply-To: References: <11a6bcc6-8fc7-0863-98de-ac071fdcc8b0@cs.otago.ac.nz> <018e93d8-a53d-c5e9-c920-57e7c7086e68@ericsson.com> <098c5d52-5435-8c77-9fad-8c8247447034@cs.otago.ac.nz> Message-ID: <628D90FD-CEBD-4678-8B99-6BABBB6108DF@gmail.com> to continue the discussion i just commited a simple library that does this: https://gitlab.com/barrel-db/ehash Hopefully such thing will be optimised soon :) - benoit > On 5 Nov 2017, at 17:44, Benoit Chesneau wrote: > > > >> On 30 Oct 2017, at 01:16, Richard A. O'Keefe wrote: >> >> For term_to_binary/2, of course the result >> depends on the Options argument, but I take >> it now that even being explicit about the >> Options is not enough. Can we have a >> 'canonical' option? >> > > +1 . That would be an awesome option :) > > - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Thu Nov 23 09:20:06 2017 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Thu, 23 Nov 2017 09:20:06 +0100 Subject: [erlang-questions] Worth minimizing the number of inter-node messages? In-Reply-To: <12F2115FD1CCEE4294943B2608A18FA301C2A10269@MAIL01.win.lbaum.eu> References: <12F2115FD1CCEE4294943B2608A18FA301C2A10269@MAIL01.win.lbaum.eu> Message-ID: <3D333879-AB9E-47D6-9D23-0E053CCDA5AB@gmail.com> Ah! Thanks! That ?Manifold? thingy sounds interesting. I?ll have a look Stefan > 23 nov. 2017 kl. 09:13 skrev Tobias Schlager : > > Hi Stefan, > > your question reminds me of a Discord blog about inter-node messaging I read, some time ago. I was lucky enough to be able to find it again. I know, it's about Elixir, but the concepts apply nonetheless. Here you go [1]. > > Regards > Tobias > > [1] https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Thu Nov 23 09:21:04 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 23 Nov 2017 10:21:04 +0200 Subject: [erlang-questions] Worth minimizing the number of inter-node messages? In-Reply-To: References: <4C760A9B-FD45-48F1-BA17-A6E1BAA42E88@gmail.com> Message-ID: <318D694B-43EB-4F0D-BA36-199F82FC1A79@gmail.com> Hello, The Erlang distribution is good for signal plane and not optimal for data plane? 1000 messages per second is less then 1MB for interprocess communication. You?ve also missed the number of nodes in your cluster. Therefore, it is not clear what is the bandwidth requirements per link. The broadcast of messages would not an optimal for highly-loaded production systems but pre-mature optimisation is an evil. * Go with broadcast as first and rapid PoC implementation. * Optimise it with group proxy to save bandwidth of inter-node links. * Go with external data plane channels (e.g. TCP) or external queue systems. - Dmitry > On 23 Nov 2017, at 10.03, Stefan Hellkvist wrote: > > Each process (or "user" if you prefer the group chat analogy) could send a message, say once every second. So if there are 1000 processes in one such "room" there are 1000 messages per second. Then there could be many such "rooms" also in the system, running on different nodes, but that is in the order of 10 at most. > > On Thu, Nov 23, 2017 at 8:55 AM, Sergej Jure?ko > wrote: > You neglected to mention a critical aspect. How often are you broadcasting? > > Regards, > Sergej > > > On 23 Nov 2017, at 08:49, Stefan Hellkvist > wrote: > > > > Hi all enlightened people, > > > > Suppose that you, in a distributed Erlang setup, have a process A, running on one of the nodes, which, in its state, holds pids B1, B2, ..., Bn, where each process B could be running on any of the nodes in the cluster. Suppose further that you want to typically broadcast messages to all processes (B1 through Bn) - the same message to all processes (you can think of it as a "group chat" if you like where "A" is the room and B1-Bn are the users joining the room - seeing everything posted there - or you could see it as a message bus for a topic "A"). The number n here could be in the order of "2-10 thousands" and messages are typically not huge but perhaps not tiny either (< 1kB). > > > > In such a setup, if you want to minimize the time to distribute the message to all processes B1 to Bn (the time between when the first B receives the message to the last), would you consider it be worth to try to minimize the amount of traffic between the nodes, by, let's say, grouping all processes belonging to the same node and instead adding such groups to the process A? Then there would be one message going between the nodes and then the local groups on each node take care of sending it to all its local processes (under the assumption that local message passing has a lower cost than inter-node message passing). Is there even such library out there that does something similar? I know of pg and pg2 but the goals of these modules is not about minimizing node-to-node traffic if I understand them correctly. > > > > Or perhaps others are approaching such broadcasting of messages in different ways that does not rely on Erlang message passing at all - such as relying on multicast or using message buses (such as Rabbit or NATS)? Perhaps that is a better approach? I suppose "measuring" is the answer here...but perhaps someone has done similar comparisons already? > > > > /Stefan > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Thu Nov 23 10:34:17 2017 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Thu, 23 Nov 2017 10:34:17 +0100 Subject: [erlang-questions] Worth minimizing the number of inter-node messages? In-Reply-To: <318D694B-43EB-4F0D-BA36-199F82FC1A79@gmail.com> References: <4C760A9B-FD45-48F1-BA17-A6E1BAA42E88@gmail.com> <318D694B-43EB-4F0D-BA36-199F82FC1A79@gmail.com> Message-ID: Thank you for your valuable input. Yes, I think that is probably the way - to take it step wise in that order that you mention and avoid optimizing before it is really needed. The number of nodes needed, as you mention, is not known at the moment. I'm currently using ONE node. This work is just to try and understand how one would scale things out if there is a need to support more traffic. /Stefan On Thu, Nov 23, 2017 at 9:21 AM, Dmitry Kolesnikov wrote: > Hello, > > The Erlang distribution is good for signal plane and not optimal for data > plane? > > 1000 messages per second is less then 1MB for interprocess communication. > You?ve also missed the number of nodes in your cluster. Therefore, it is > not clear what is the bandwidth requirements per link. > > The broadcast of messages would not an optimal for highly-loaded > production systems but pre-mature optimisation is an evil. > > * Go with broadcast as first and rapid PoC implementation. > * Optimise it with group proxy to save bandwidth of inter-node links. > * Go with external data plane channels (e.g. TCP) or external queue > systems. > > - Dmitry > > > > On 23 Nov 2017, at 10.03, Stefan Hellkvist wrote: > > Each process (or "user" if you prefer the group chat analogy) could send a > message, say once every second. So if there are 1000 processes in one such > "room" there are 1000 messages per second. Then there could be many such > "rooms" also in the system, running on different nodes, but that is in the > order of 10 at most. > > On Thu, Nov 23, 2017 at 8:55 AM, Sergej Jure?ko > wrote: > >> You neglected to mention a critical aspect. How often are you >> broadcasting? >> >> Regards, >> Sergej >> >> > On 23 Nov 2017, at 08:49, Stefan Hellkvist wrote: >> > >> > Hi all enlightened people, >> > >> > Suppose that you, in a distributed Erlang setup, have a process A, >> running on one of the nodes, which, in its state, holds pids B1, B2, ..., >> Bn, where each process B could be running on any of the nodes in the >> cluster. Suppose further that you want to typically broadcast messages to >> all processes (B1 through Bn) - the same message to all processes (you can >> think of it as a "group chat" if you like where "A" is the room and B1-Bn >> are the users joining the room - seeing everything posted there - or you >> could see it as a message bus for a topic "A"). The number n here could be >> in the order of "2-10 thousands" and messages are typically not huge but >> perhaps not tiny either (< 1kB). >> > >> > In such a setup, if you want to minimize the time to distribute the >> message to all processes B1 to Bn (the time between when the first B >> receives the message to the last), would you consider it be worth to try to >> minimize the amount of traffic between the nodes, by, let's say, grouping >> all processes belonging to the same node and instead adding such groups to >> the process A? Then there would be one message going between the nodes and >> then the local groups on each node take care of sending it to all its local >> processes (under the assumption that local message passing has a lower cost >> than inter-node message passing). Is there even such library out there that >> does something similar? I know of pg and pg2 but the goals of these modules >> is not about minimizing node-to-node traffic if I understand them correctly. >> > >> > Or perhaps others are approaching such broadcasting of messages in >> different ways that does not rely on Erlang message passing at all - such >> as relying on multicast or using message buses (such as Rabbit or NATS)? >> Perhaps that is a better approach? I suppose "measuring" is the answer >> here...but perhaps someone has done similar comparisons already? >> > >> > /Stefan >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunp@REDACTED Thu Nov 23 12:22:01 2017 From: arunp@REDACTED (Arun) Date: Thu, 23 Nov 2017 16:52:01 +0530 Subject: [erlang-questions] Mnesia deleting log file In-Reply-To: References: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> Message-ID: <394a0fc5-9857-14ef-f032-f5267dfe4627@utl.in> Even after setting the "dump_log_update_in_place" configuration to false, the data is not getting retained after a power outage. Any more configurations that can be tried. Regards, Arun P On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote: > > I agree?setting dump_log_in_place to false makes extremely unlikely > for mnesia to fail in this way. Both JFFS2 and UBIFS ensure that > rename is atomic, even when power is pulled. > > For desktop applications, the culprit is likely the VM being killed, > not the whole system crashing. Thus the rename was either executed or > not; either way we have a complete DAT file. > > There is always a chance of things like this happening: > http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but > this shouldn?t happen on most hardware. > > Note that the default for dump_log_time_threshold is 3 minutes. Speed > vs Safety is always a trade-off, but many people will be surprised to > lose data in a crash 2 minutes after saving. > > *From:*Jesper Louis Andersen [mailto:jesper.louis.andersen@REDACTED] > *Sent:* Wednesday, November 22, 2017 7:54 AM > *To:* Michael Schmidt > *Cc:* Arun ; erlang-questions@REDACTED > *Subject:* Re: [erlang-questions] Mnesia deleting log file > > This thread makes we wonder about the following section in the Mnesia > documentation: > > "If a power failure occurs during the dump, this can cause the > randomly accessed DAT files to become corrupt. If the parameter is set > to false, Mnesia copies the DAT files and target the dump to the new > temporary files. If the dump is successful, the temporary files are > renamed to their normal DAT suffixes. The possibility for > unrecoverable inconsistencies in the data files becomes much smaller > with this strategy. However, the actual dumping of the transaction log > becomes considerably slower. The system designer must decide whether > speed or safety is the higher priority." > > In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an > OS crash might leave you with a temporary file and the target file, > hardlinked as if ln() was called, but the move itself is atomic, even > under OS crashes. Of course, this assumes you correctly wrote your > temporary file, fsync()'ed the file and then performed the rename() as > you should. > > With that information in mind, what is the scenario which can make > mnesia fail if the dump_log_update_in_place is set to false? There may > be a window, but the operating system cannot be the provider of this > window at all, unless it is incorrectly implemented or configured[1]. > Hence my question. > > [1] The primary concern are write caches on disks which can have a > write hole unless they have proper (working!) battery backup of their > cache or are able to write their memory to a small solid state part of > the chip. Or if you are using ZFS where copy-on-write semantics > completely eliminates the write hole. > > On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt > > wrote: > > Hi Arun, > > If you happen to reboot when Mnesia is writing out its logfile, it > can become corrupt (and lose data). We fight this issue on > embedded devices a lot. > > Look ?Configuration Parameters? section here: > > http://erlang.org/doc/man/mnesia.html#id69584 > > You will want to set dump_log_update_in_placeto false. This will > make it write out the table to a new file, and rename it the very > end. > > One other option to look at is dump_log_time_thresholdto change > how often items are flushed to disk. For configuration data, you > can likely set this to a few seconds. > > These options can be set via the sys.config file as well > > Mike > > *From:*erlang-questions-bounces@REDACTED > [mailto:erlang-questions-bounces@REDACTED > ] *On Behalf Of *Arun > *Sent:* Tuesday, November 21, 2017 10:16 PM > *To:* erlang-questions@REDACTED > *Subject:* [erlang-questions] Mnesia deleting log file > > Dear all, > > I've a program written in erlang which uses Mnesia application as > the database application. I've a table created by name > "configuration_table" which stores > > certain configurations that need to be persistent. Occasionally, > whenever I restart my program the following error is thrown by > mnesia and I end up losing > > all the persistent configurations. > > I've searched about this problem in the erlang documentation and > all it tells is "Node not running". What could be the probable > cause for this problem > > and how do I fix it? > > Mnesia('test@REDACTED '): Data may > be missing, Corrupt logfile deleted: > "/home/utl/mnesia_database/configuration_table.DCL", > {node_not_running, > 'test@REDACTED '} > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud > service. > ______________________________________________________________________ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > ______________________________________________________________________ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Nov 23 12:46:46 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 23 Nov 2017 12:46:46 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <20171123024552.GD2024@ferdmbp.local> References: <1511379790.16757707@apps.rackspace.com> <20171123024552.GD2024@ferdmbp.local> Message-ID: <0d0c30db-0a43-10ea-9536-4265f10437de@ninenines.eu> On 11/23/2017 03:45 AM, Fred Hebert wrote: > But overall, you can't escape Unicode. Work with JSON? There it is. > HTTP? There again. URLs? You bet. File systems? Hell yes! Erlang modules > and app files: also yes! Nitpick but... HTTP itself is ASCII. There used to be support for characters in the 128..255 range but they are now obsolete. URLs with Unicode must be encoded. Headers must be ASCII, if you need Unicode there's https://tools.ietf.org/html/rfc8187 to the rescue, again via urlencoding. My point is, you can go very far with HTTP without ever bothering with Unicode. And it seems like this will be true in the foreseeable future, as the upcoming httptre is mostly clarifications to the current httpbis specifications in preparation for QUIC. I suspect the same is true for most of the older protocols. And I would say that's a good thing as it considerably reduces the complexity of the protocol. Of course applications themselves are another matter entirely, and if you deal with anything you haven't written yourself, chances are you will need to deal with Unicode. -- Lo?c Hoguin https://ninenines.eu From n.oxyde@REDACTED Thu Nov 23 13:20:37 2017 From: n.oxyde@REDACTED (Anthony Ramine) Date: Thu, 23 Nov 2017 13:20:37 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511379790.16757707@apps.rackspace.com> References: <1511379790.16757707@apps.rackspace.com> Message-ID: > Le 22 nov. 2017 ? 20:43, lloyd@REDACTED a ?crit : > > Dear Gods of Erlang, > > "This module has been reworked in Erlang/OTP 20 to handle unicode:chardata() and operate on grapheme clusters. The old functions that only work on Latin-1 lists as input are still available but should not be used. They will be deprecated in Erlang/OTP 21." > > I'm sorry. I've brought up this issue before and got lots of push back. > > But every time I look up tried and true and long-used string functions to find that they are deprecated and will be dropped in future Erlang releases my blood pressure soars. Both my wife and my doctor tell me that at my age this is a dangerous thing. > > I do understand the importance and necessity of Unicode. And applaud the addition of Unicode functions. > > But the deprecated string functions have a long history. The English language and Latin-1 characters are widely used around the world. You do know that Latin-1 cannot be used to represent all English words, right? > Yes, it should be easy for programmers to translate code from one user language to another. But I'm not convinced that the Gods of Erlang have found the optimal solution by dropping all Latin-1 string functions. > > My particular application is directed toward English speakers. So, until further notice, I have no use for Unicode. Damn, I hope your users will never want to tell their friends how delicious was the hors-d'?uvre they ate yesterday. > I don't want to sound like nationalist pig, but I think dropping the Latin-1 string functions from future Erlang releases is a BIG mistake. > > I look up tokens/2, a function that I use fairly frequently, and I see that it's deprecated. I look up the suggested replacement and I see lexemes/2. > > So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I see that a lexeme is "a meaningful linguistic unit." > > Meaning what? I just want to turn "this and that" into "This And That." > > I read further in the Erlang docs and I see "grapheme cluster." WHAT THE ... IS GRAPHEME CLUSTER? > > I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a unit of a writing system." > > Ah yes, grapheme is defined in the docs. But I have to read and re-read the definition to understand what the God's of Erlang mean by a "graphene cluster." And I'm still not sure I get it. > > It sounds like someone took a linguistics class and is trying to show off. > > But now I've spent 30 minutes--- time that I don't have to waste trying to figure out how do a simple manipulation of "this and that." Recurse the next time I want to look up a string function in the Erlang docs. IMO the functions should have been named according to "grapheme cluster", not "lexeme". > SOLUTION > > Keep the Latin-1 string functions. Put them in a separate library if necessary. Or put the new Unicode functions in a separate library. But don't arbitrarily drop them. > > Some folks have suggested that I maintain my own library of the deprecated Latin1 functions. But why should I have to do that? How does that help other folks with the same issue? The issue is that you want to keep using Latin-1 (which Latin-1 btw, you do know there are at least 2 of them? Do you know which one Erlang uses? Beware that's a tricky question) instead of switching to Unicode, which will benefit even your English users. > Bottom line: please please please do not drop the existing Latin-1 string functions. > > Please don't. > > Best wishes, > > LRP From pepm.workshop@REDACTED Thu Nov 23 13:06:11 2017 From: pepm.workshop@REDACTED (PEPM Workshop) Date: Thu, 23 Nov 2017 21:06:11 +0900 Subject: [erlang-questions] PEPM 2018 Call for Poster/Demo Abstracts and Participation Message-ID: -- Call for Poster/Demo Abstracts and Participation -- ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation (PEPM) 2018 =============================================================================== * Website : http://popl18.sigplan.org/track/PEPM-2018 * Time : 8th ? 9th January 2018 * Place : Los Angeles, CA, US (co-located with POPL 2018) POSTER/DEMO SESSIONS: PEPM 2018 is accepting proposals for poster/demo presentations on a rolling basis, until 8th December (AoE). See below for the submission guidelines. Registration ------------ * Web page : https://popl18.sigplan.org/attending/Registration * Early registration deadline : 10th December 2017 Invited speakers ---------------- Alex Aiken (Stanford University) Conal Elliott (Target) Jan Midtgaard (University of Southern Denmark) Accepted papers --------------- * https://popl18.sigplan.org/track/PEPM-2018#event-overview A Guess-and-Assume Approach to Loop Fusion for Program Verification Akifumi Imanishi, Kohei Suenaga, and Atsushi Igarashi Checking Cryptographic API Usage with Composable Annotations (Short Paper) Duncan Mitchell, L. Thomas van Binsbergen, Blake Loring, and Johannes Kinder Gradually Typed Symbolic Expressions David Broman and Jeremy G. Siek On the Cost of Type-Tag Soundness Ben Greenman and Zeina migeed Partially Static Data as Free Extension of Algebras (Short Paper) Jeremy Yallop, Tamara von Glehn, and Ohad Kammar Program Generation for ML Modules (Short Paper) Takahisa Watanabe and Yukiyoshi Kameyama Recursive Programs in Normal Form (Short Paper) Barry Jay Selective CPS Transformation for Shift and Reset Kenichi Asai and Chihiro Uehara Poster/demo abstract submission guideline ----------------------------------------- * https://popl18.sigplan.org/track/PEPM-2018#Call-for-Poster-Demo-Abstracts To maintain PEPM?s dynamic and interactive nature, PEPM 2018 will continue to have special sessions for poster/demo presentations. In addition to the main interactive poster/demo session, there will also be a scheduled short-talk session where each poster/demo can be advertised to the audience in, say, 5?10 minutes. Poster/demo abstracts should describe work relevant to PEPM (whose scope is detailed below), typeset as a one-page PDF using the two-column ?sigplan? sub-format of the new ?acmart? format available at: http://sigplan.org/Resources/Author/ and sent by email to the programme co-chairs, Fritz Henglein and Josh Ko, at: henglein@REDACTED, hsiang-shang@REDACTED Please also include in the email: * a short summary of the abstract (in plain text), * the type(s) of proposed presentation (poster and/or demo), and * whether you would like to give a scheduled short talk (in addition to the poster/demo presentation). Abstracts should be sent no later than: Friday, 8th December 2017, anywhere on earth and will be considered for acceptance on a rolling basis. Accepted abstracts, along with their short summary, will be posted on PEPM 2018?s website. At least one author of each accepted abstract must attend the workshop and present the work during the poster/demo session. Student participants with accepted posters/demos can apply for a SIGPLAN PAC grant to help cover travel expenses and other support. PAC also offers other support, such as for child-care expenses during the meeting or for travel costs for companions of SIGPLAN members with physical disabilities, as well as for travel from locations outside of North America and Europe. For details on the PAC programme, see its web page. Scope ----- In addition to the traditional PEPM topics (see below), PEPM 2018 welcomes submissions in new domains, in particular: * Semantics based and machine-learning based program synthesis and program optimisation. * Modelling, analysis, and transformation techniques for distributed and concurrent protocols and programs, such as session types, linear types, and contract specifications. More generally, topics of interest for PEPM 2018 include, but are not limited to: * Program and model manipulation techniques such as: supercompilation, partial evaluation, fusion, on-the-fly program adaptation, active libraries, program inversion, slicing, symbolic execution, refactoring, decompilation, and obfuscation. * Techniques that treat programs/models as data objects including metaprogramming, generative programming, embedded domain-specific languages, program synthesis by sketching and inductive programming, staged computation, and model-driven program generation and transformation. * Program analysis techniques that are used to drive program/model manipulation such as: abstract interpretation, termination checking, binding-time analysis, constraint solving, type systems, automated testing and test case generation. * Application of the above techniques including case studies of program manipulation in real-world (industrial, open-source) projects and software development processes, descriptions of robust tools capable of effectively handling realistic applications, benchmarking. Examples of application domains include legacy program understanding and transformation, DSL implementations, visual languages and end-user programming, scientific computing, middleware frameworks and infrastructure needed for distributed and web-based applications, embedded and resource-limited computation, and security. This list of categories is not exhaustive, and we encourage submissions describing new theories and applications related to semantics-based program manipulation in general. If you have a question as to whether a potential submission is within the scope of the workshop, please contact the programme co-chairs, Fritz Henglein and Josh Ko (henglein@REDACTED, hsiang-shang@REDACTED). From jesper.louis.andersen@REDACTED Thu Nov 23 15:21:04 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 23 Nov 2017 14:21:04 +0000 Subject: [erlang-questions] Mnesia deleting log file In-Reply-To: <394a0fc5-9857-14ef-f032-f5267dfe4627@utl.in> References: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> <394a0fc5-9857-14ef-f032-f5267dfe4627@utl.in> Message-ID: I'm curious: are the data written as part of a mnesia transaction? On Thu, Nov 23, 2017 at 12:22 PM Arun wrote: > Even after setting the "dump_log_update_in_place" configuration to false, > the data is not getting retained > after a power outage. Any more configurations that can be tried. > Regards, > Arun P > > > > On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote: > > I agree?setting dump_log_in_place to false makes extremely unlikely for > mnesia to fail in this way. Both JFFS2 and UBIFS ensure that rename is > atomic, even when power is pulled. > > > > For desktop applications, the culprit is likely the VM being killed, not > the whole system crashing. Thus the rename was either executed or not; > either way we have a complete DAT file. > > > > There is always a chance of things like this happening: > http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but > this shouldn?t happen on most hardware. > > > > Note that the default for dump_log_time_threshold is 3 minutes. Speed vs > Safety is always a trade-off, but many people will be surprised to lose > data in a crash 2 minutes after saving. > > > > > > *From:* Jesper Louis Andersen [mailto:jesper.louis.andersen@REDACTED > ] > *Sent:* Wednesday, November 22, 2017 7:54 AM > *To:* Michael Schmidt > > *Cc:* Arun ; erlang-questions@REDACTED > *Subject:* Re: [erlang-questions] Mnesia deleting log file > > > > This thread makes we wonder about the following section in the Mnesia > documentation: > > "If a power failure occurs during the dump, this can cause the randomly > accessed DAT files to become corrupt. If the parameter is set to false, > Mnesia copies the DAT files and target the dump to the new temporary > files. If the dump is successful, the temporary files are renamed to their > normal DAT suffixes. The possibility for unrecoverable inconsistencies in > the data files becomes much smaller with this strategy. However, the actual > dumping of the transaction log becomes considerably slower. The system > designer must decide whether speed or safety is the higher priority." > > In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS > crash might leave you with a temporary file and the target file, hardlinked > as if ln() was called, but the move itself is atomic, even under OS > crashes. Of course, this assumes you correctly wrote your temporary file, > fsync()'ed the file and then performed the rename() as you should. > > With that information in mind, what is the scenario which can make mnesia > fail if the dump_log_update_in_place is set to false? There may be a > window, but the operating system cannot be the provider of this window at > all, unless it is incorrectly implemented or configured[1]. Hence my > question. > > [1] The primary concern are write caches on disks which can have a write > hole unless they have proper (working!) battery backup of their cache or > are able to write their memory to a small solid state part of the chip. Or > if you are using ZFS where copy-on-write semantics completely eliminates > the write hole. > > > > > > On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt < > Michael.K.Schmidt@REDACTED> wrote: > > Hi Arun, > > > > If you happen to reboot when Mnesia is writing out its logfile, it can > become corrupt (and lose data). We fight this issue on embedded devices a > lot. > > > > Look ?Configuration Parameters? section here: > > http://erlang.org/doc/man/mnesia.html#id69584 > > > > You will want to set dump_log_update_in_place to false. This will make > it write out the table to a new file, and rename it the very end. > > > > One other option to look at is dump_log_time_threshold to change how > often items are flushed to disk. For configuration data, you can likely > set this to a few seconds. > > > > These options can be set via the sys.config file as well > > > > Mike > > > > *From:* erlang-questions-bounces@REDACTED [mailto: > erlang-questions-bounces@REDACTED] *On Behalf Of *Arun > *Sent:* Tuesday, November 21, 2017 10:16 PM > *To:* erlang-questions@REDACTED > *Subject:* [erlang-questions] Mnesia deleting log file > > > > Dear all, > > I've a program written in erlang which uses Mnesia application as the > database application. I've a table created by name "configuration_table" > which stores > > certain configurations that need to be persistent. Occasionally, whenever > I restart my program the following error is thrown by mnesia and I end up > losing > > all the persistent configurations. > > I've searched about this problem in the erlang documentation and all it > tells is "Node not running". What could be the probable cause for this > problem > > and how do I fix it? > > Mnesia('test@REDACTED '): Data may be missing, > Corrupt logfile deleted: > "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running, > 'test@REDACTED '} > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > ______________________________________________________________________ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > ______________________________________________________________________ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Thu Nov 23 15:22:15 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 23 Nov 2017 14:22:15 +0000 Subject: [erlang-questions] C-Node breakage with git master In-Reply-To: <1872734591.3088063.1511285464784.JavaMail.zimbra@tpip.net> References: <767319448.3087859.1511277816985.JavaMail.zimbra@tpip.net> <87813715.3087874.1511278298477.JavaMail.zimbra@tpip.net> <1511284777.15708.52.camel@ericsson.com> <1872734591.3088063.1511285464784.JavaMail.zimbra@tpip.net> Message-ID: <1511446935.3545.9.camel@ericsson.com> I've pushed a fix to master now. https://github.com/erlang/otp/commit/8ed0d75c186d9da24bd6cfb85732487b17 a3b054 /Sverker On tis, 2017-11-21 at 18:31 +0100, Andreas Schultz wrote: > Hi, > > ----- On Nov 21, 2017, at 6:19 PM, Sverker Eriksson sverker.eriksson@ > ericsson.com wrote: > > > > > Hi Andreas > > > > Current master is broken with respect to monitor of c-node > > "processes". > Good to know ;-) > ? > > > > I'm currently schratching my head trying to fix this mess. Nice to > > have > > a shoulder to cry on. > > > > The commit?https://github.com/erlang/otp/commit/17e198d6ee60f7dec9a > > bfed > > 272cf4226?is part of a work to make distributed operations (like > > send, > > monitor, link, etc) to *not* wait for the connections setup to > > complete. They should just enqueue their request and trigger a > > connection setup to commence. > > > > This can be done in a backward compatible manner as all of them > > have a > > truly asynchronous interface. > > > > All except erlang:monitor, when called toward c-nodes which do not > > support process monitoring. Old behaviour is to throw badarg. But > > that > > would force erlang:monitor to still be synchronous waiting for > > connection setup in order to know if the node supports it or not. > I would be nice if we could have to way to support monitors in a > C-node as well. It would have to be the responsibility of the > C-node to implement whatehttps://github.com/erlang/otp/commit/8ed0d75 > c186d9da24bd6cfb85732487b17a3b054ver kind of monitoring it sees fit > when such > a request comes in. > > And since I'm dreaming, cached atoms in C-nodes would be great ;-) > > > > > So the idea is to slightly change the behavoir and instead of > > badarg go > > ahead and create the monitor but only let it supervise the > > connection. > > That is, you will only get 'DOWN' message with 'noconnection' from > > such > > a monitor. This is similar to what gen:call implements today by > > cathing > > badarg and using monitor_node. > Sounds good to me. > > Andreas > > > > > Yes, erl_interface sets DFLAG_DIST_MONITOR but it does not > > implement > > it. Why? I think it's an old sloppy fix to > > make?erl_global_register() > > work, where the c-node receives a monitor request as part of the > > reply > > which it just ignores. > > > > > > > > /Sverker > > > > > > > > On tis, 2017-11-21 at 16:31 +0100, Andreas Schultz wrote: > > > > > > ----- On Nov 21, 2017, at 4:23 PM, Andreas Schultz aschultz@REDACTED > > > net > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > Something in the C-node handling changed between otp-20.1.3 and > > > > today git master branch. > > > Did sent to early, the behavior was changed by this commit: > > > > > > https://github.com/erlang/otp/commit/17e198d6ee60f7dec9abfed272cf > > > 4226 > > > aea44535 > > > > > > I think the removal of the DFLAG_DIST_MONITOR and > > > DFLAG_DIST_MONITOR_NAME > > > in that cset is a mistake. > > > > > > Also, the removed DFLAG_DIST_MONITOR test proves IMHO that a > > > C-Node should not set that flag. > > > > > > Comments? > > > > > > Regards > > > Andreas > > > > > > > > > > > > > > > > > > > I have a C-node that implements the remote end of > > > > net_adm:ping/1. > > > > To do that, it needs to answer to > > > > > > > > ?gen:call({net_kernel, Node}, '$gen_call', {is_auth, node()}, > > > > infinity) > > > > > > > > Code is at [1]. > > > > > > > > In 20.1.3, the call is send as a plain message. With today's > > > > git > > > > master > > > > it first attempts to setup a monitor (a ERL_MONITOR_P message > > > > is > > > > sent). > > > > > > > > Monitors are not supported by C-node and ei_xreceive_msg > > > > therefor > > > > returns > > > > an error, causing a connection abort. > > > > > > > > Reading through some the distribution code and gen.erl code, it > > > > seems > > > > that the C-node support is somewhat broken to begin with. There > > > > is > > > > a comment in gen.erl [2] that suggests that a attempting to > > > > setup a > > > > monitor on a C-node should return with an error. However, this > > > > would > > > > need to be done on sending side. ei_xreceive_msg has no support > > > > to > > > > deal with it and the error code would not allow the consumer to > > > > implement proper handling. > > > > > > > > There are two flags in the distribution protocol > > > > (DFLAG_DIST_MONITOR and > > > > DFLAG_DIST_MONITOR_NAME). C-Nodes do set the > > > > DFLAG_DIST_MONITOR, > > > > but not > > > > the DFLAG_DIST_MONITOR_NAME flag. This seems to be wrong, IMHO > > > > the > > > > DFLAG_DIST_MONITOR should be cleared. > > > > > > > > But, I also can't find the place where erlang:monitor/1 would > > > > actually > > > > check and honor those flags. The monitor BIF seems to always > > > > send a > > > > monitor request regardless of the node flags. > > > > > > > > So my questions are: > > > > > > > > 1. Has anyone an idea what changed to cause the ping/is_auth > > > > change? > > > > 2. What is the correct way to implement a gen_server in a C- > > > > Node or > > > > ? is it in deed currently not possible/broken? > > > > > > > > Regards > > > > Andreas > > > > > > > > [1]: https://github.com/travelping/capwap-dp/blob/master/src/ca > > > > pwap > > > > -dp.c#L943 > > > > [2]: https://github.com/erlang/otp/blob/master/lib/stdlib/src/g > > > > en.e > > > > rl#L184 > > > > -- > > > > Dipl.-Inform. Andreas Schultz > > > > > > > > email: as@REDACTED > > > > phone: +49-391-819099-224 > > > > > > > > ----------------------- enabling your networks ---------------- > > > > ---- > > > > -- > > > > > > > > Travelping GmbH?????????????????????phone:??+49-391-81 90 99 0 > > > > Roentgenstr.??13????????????????????fax:????+49-391-81 90 99 > > > > 299 > > > > 39108 Magdeburg?????????????????????email:??info@REDACTED > > > > GERMANY?????????????????????????????web:????http://www.travelpi > > > > ng.c > > > > om > > > > > > > > Company Registration: Amtsgericht Stendal????????Reg No.:???HRB > > > > 10578 > > > > Geschaeftsfuehrer: Holger Winkelmann??????????VAT ID No.: > > > > DE236673780 > > > > ------------------------------------------------------------- > > > > ---- > > > > ---- > > > > _______________________________________________ > > > > erlang-questions mailing list > > > > erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions From Ingela.Anderton.Andin@REDACTED Thu Nov 23 16:26:39 2017 From: Ingela.Anderton.Andin@REDACTED (Ingela Anderton Andin) Date: Thu, 23 Nov 2017 16:26:39 +0100 Subject: [erlang-questions] Patch Package: OTP 20.1.7 Message-ID: <652d176b-19dd-f96e-fce9-e68744540759@ericsson.com> Patch Package:??? OTP 20.1.7 Git Tag:???????????????? OTP-20.1.7 Date:??????????????????? 2017-11-22 Trouble Report Id:? OTP-14632, OTP-14653, OTP-14655, OTP-14748, OTP-14766 Seq num: System:????????????????? OTP Release:???????????????? 20 Application:??????????? public_key-1.5.1, ssl-8.2.2 Predecessor:????????? OTP 20.1.6 ?Check out the git tag OTP-20.1.7, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- public_key-1.5.1 ------------------------------------------------ ?--------------------------------------------------------------------- ?The public_key-1.5.1 application can be applied independently of ?other applications on a full OTP 20 installation. ?--- Improvements and New Features --- ? OTP-14653??? Application(s): public_key ?????????????? Hostname verification: Add handling of the general name ?????????????? iPAddress in certificate's subject alternative name ?????????????? extension (subjAltName). ? OTP-14766??? Application(s): public_key ?????????????? Correct key handling in pkix_test_data/1 and use a ?????????????? generic example mail address instead of an existing ?????????????? one. ?Full runtime dependencies of public_key-1.5.1: asn1-3.0, crypto-3.8, ?erts-6.0, kernel-3.0, stdlib-2.0 ?--------------------------------------------------------------------- ?--- ssl-8.2.2 ------------------------------------------------------- ?--------------------------------------------------------------------- ?Note! The ssl-8.2.2 application can *not* be applied independently of ?????? other applications on an arbitrary OTP 20 installation. ?????? On a full OTP 20 installation, also the following runtime ?????? dependency has to be satisfied: ?????? -- public_key-1.5 (first satisfied in OTP 20.1) ?--- Fixed Bugs and Malfunctions --- ? OTP-14632??? Application(s): ssl ?????????????? TLS sessions must be registered with SNI if provided, ?????????????? so that sessions where client hostname verification ?????????????? would fail can not connect reusing a session created ?????????????? when the server name verification succeeded. ?????????????? Thanks to Graham Christensen for reporting this. ? OTP-14748??? Application(s): ssl ?????????????? An erlang TLS server configured with cipher suites ?????????????? using rsa key exchange, may be vulnerable to an ?????????????? Adaptive Chosen Ciphertext attack (AKA Bleichenbacher ?????????????? attack) against RSA, which when exploited, may result ?????????????? in plaintext recovery of encrypted messages and/or a ?????????????? Man-in-the-middle (MiTM) attack, despite the attacker ?????????????? not having gained access to the server?s private key ?????????????? itself. CVE-2017-1000385 ?????????????? Exploiting this vulnerability to perform plaintext ?????????????? recovery of encrypted messages will, in most practical ?????????????? cases, allow an attacker to read the plaintext only ?????????????? after the session has completed. Only TLS sessions ?????????????? established using RSA key exchange are vulnerable to ?????????????? this attack. ?????????????? Exploiting this vulnerability to conduct a MiTM attack ?????????????? requires the attacker to complete the initial attack, ?????????????? which may require thousands of server requests, during ?????????????? the handshake phase of the targeted session within the ?????????????? window of the configured handshake timeout. This attack ?????????????? may be conducted against any TLS session using RSA ?????????????? signatures, but only if cipher suites using RSA key ?????????????? exchange are also enabled on the server. The limited ?????????????? window of opportunity, limitations in bandwidth, and ?????????????? latency make this attack significantly more difficult ?????????????? to execute. ?????????????? RSA key exchange is enabled by default although least ?????????????? prioritized if server order is honored. For such a ?????????????? cipher suite to be chosen it must also be supported by ?????????????? the client and probably the only shared cipher suite. ?????????????? Captured TLS sessions encrypted with ephemeral cipher ?????????????? suites (DHE or ECDHE) are not at risk for subsequent ?????????????? decryption due to this vulnerability. ?????????????? As a workaround if default cipher suite configuration ?????????????? was used you can configure the server to not use ?????????????? vulnerable suites with the ciphers option like this: ?????????????? {ciphers, [Suite || Suite <- ssl:cipher_suites(), ?????????????? element(1,Suite) =/= rsa]} ?????????????? that is your code will look somethingh like this: ?????????????? ssl:listen(Port, [{ciphers, [Suite || Suite <- ?????????????? ssl:cipher_suites(), element(1,S) =/= rsa]} | ?????????????? Options]). ?????????????? Thanks to Hanno B?ck, Juraj Somorovsky and Craig Young ?????????????? for reporting this vulnerability. ?--- Improvements and New Features --- ? OTP-14655??? Application(s): ssl ?????????????? If no SNI is available and the hostname is an ?????????????? IP-address also check for IP-address match. This check ?????????????? is not as good as a DNS hostname check and certificates ?????????????? using IP-address are not recommended. ?????????????? Thanks to Graham Christensen for reporting this. ?Full runtime dependencies of ssl-8.2.2: crypto-3.3, erts-7.0, ?inets-5.10.7, kernel-3.0, public_key-1.5, stdlib-3.2 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ingela.Anderton.Andin@REDACTED Thu Nov 23 16:26:55 2017 From: Ingela.Anderton.Andin@REDACTED (Ingela Anderton Andin) Date: Thu, 23 Nov 2017 16:26:55 +0100 Subject: [erlang-questions] Patch Package: OTP 19.3.6.4 Message-ID: <0a953d81-6984-e875-211f-990dd3dd7495@ericsson.com> Patch Package:??? OTP 19.3.6.4 Git Tag:??????????????? OTP-19.3.6.4 Date:??????????????????? 2017-11-23 Trouble Report Id:? OTP-14748 Seq num: System:????????????????? OTP Release:???????????????? 19 Application:??????????? ssl-8.1.3.1 Predecessor:????????? OTP 19.3.6.3 ?Check out the git tag OTP-19.3.6.4, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- ssl-8.1.3.1 ----------------------------------------------------- ?--------------------------------------------------------------------- ?Note! The ssl-8.1.3.1 application can *not* be applied independently ?????? of other applications on an arbitrary OTP 19 installation. ?????? On a full OTP 19 installation, also the following runtime ?????? dependency has to be satisfied: ?????? -- stdlib-3.2 (first satisfied in OTP 19.2) ?--- Fixed Bugs and Malfunctions --- ? OTP-14748??? Application(s): ssl ?????????????? An erlang TLS server configured with cipher suites ?????????????? using rsa key exchange, may be vulnerable to an ?????????????? Adaptive Chosen Ciphertext attack (AKA Bleichenbacher ?????????????? attack) against RSA, which when exploited, may result ?????????????? in plaintext recovery of encrypted messages and/or a ?????????????? Man-in-the-middle (MiTM) attack, despite the attacker ?????????????? not having gained access to the server?s private key ?????????????? itself. CVE-2017-1000385 ?????????????? Exploiting this vulnerability to perform plaintext ?????????????? recovery of encrypted messages will, in most practical ?????????????? cases, allow an attacker to read the plaintext only ?????????????? after the session has completed. Only TLS sessions ?????????????? established using RSA key exchange are vulnerable to ?????????????? this attack. ?????????????? Exploiting this vulnerability to conduct a MiTM attack ?????????????? requires the attacker to complete the initial attack, ?????????????? which may require thousands of server requests, during ?????????????? the handshake phase of the targeted session within the ?????????????? window of the configured handshake timeout. This attack ?????????????? may be conducted against any TLS session using RSA ?????????????? signatures, but only if cipher suites using RSA key ?????????????? exchange are also enabled on the server. The limited ?????????????? window of opportunity, limitations in bandwidth, and ?????????????? latency make this attack significantly more difficult ?????????????? to execute. ?????????????? RSA key exchange is enabled by default although least ?????????????? prioritized if server order is honored. For such a ?????????????? cipher suite to be chosen it must also be supported by ?????????????? the client and probably the only shared cipher suite. ?????????????? Captured TLS sessions encrypted with ephemeral cipher ?????????????? suites (DHE or ECDHE) are not at risk for subsequent ?????????????? decryption due to this vulnerability. ?????????????? As a workaround if default cipher suite configuration ?????????????? was used you can configure the server to not use ?????????????? vulnerable suites with the ciphers option like this: ?????????????? {ciphers, [Suite || Suite <- ssl:cipher_suites(), ?????????????? element(1,Suite) =/= rsa]} ?????????????? that is your code will look somethingh like this: ?????????????? ssl:listen(Port, [{ciphers, [Suite || Suite <- ?????????????? ssl:cipher_suites(), element(1,S) =/= rsa]} | ?????????????? Options]). ?????????????? Thanks to Hanno B?ck, Juraj Somorovsky and Craig Young ?????????????? for reporting this vulnerability. ?Full runtime dependencies of ssl-8.1.3.1: crypto-3.3, erts-7.0, ?inets-5.10.7, kernel-3.0, public_key-1.2, stdlib-3.2 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ingela.Anderton.Andin@REDACTED Thu Nov 23 16:27:12 2017 From: Ingela.Anderton.Andin@REDACTED (Ingela Anderton Andin) Date: Thu, 23 Nov 2017 16:27:12 +0100 Subject: [erlang-questions] Patch Package: OTP 18.3.4.7 Message-ID: <9927adc9-209b-c835-6c90-e07c90999c92@ericsson.com> atch Package:???? OTP 18.3.4.7 Git Tag:???????????????? OTP-18.3.4.7 Date:??????????????????? 2017-11-23 Trouble Report Id:? OTP-14748 Seq num: System:????????????????? OTP Release:???????????????? 18 Application:??????????? ssl-7.3.3.2 Predecessor:????????? OTP 18.3.4.6 ?Check out the git tag OTP-18.3.4.7, and build a full OTP system ?including documentation. Apply one or more applications from this ?build as patches to your installation using the 'otp_patch_apply' ?tool. For information on install requirements, see descriptions for ?each application version below. ?--------------------------------------------------------------------- ?--- ssl-7.3.3.2 ----------------------------------------------------- ?--------------------------------------------------------------------- ?The ssl-7.3.3.2 application can be applied independently of other ?applications on a full OTP 18 installation. ?--- Fixed Bugs and Malfunctions --- ? OTP-14748??? Application(s): ssl ?????????????? An erlang TLS server configured with cipher suites ?????????????? using rsa key exchange, may be vulnerable to an ?????????????? Adaptive Chosen Ciphertext attack (AKA Bleichenbacher ?????????????? attack) against RSA, which when exploited, may result ?????????????? in plaintext recovery of encrypted messages and/or a ?????????????? Man-in-the-middle (MiTM) attack, despite the attacker ?????????????? not having gained access to the server?s private key ?????????????? itself. CVE-2017-1000385 ?????????????? Exploiting this vulnerability to perform plaintext ?????????????? recovery of encrypted messages will, in most practical ?????????????? cases, allow an attacker to read the plaintext only ?????????????? after the session has completed. Only TLS sessions ?????????????? established using RSA key exchange are vulnerable to ?????????????? this attack. ?????????????? Exploiting this vulnerability to conduct a MiTM attack ?????????????? requires the attacker to complete the initial attack, ?????????????? which may require thousands of server requests, during ?????????????? the handshake phase of the targeted session within the ?????????????? window of the configured handshake timeout. This attack ?????????????? may be conducted against any TLS session using RSA ?????????????? signatures, but only if cipher suites using RSA key ?????????????? exchange are also enabled on the server. The limited ?????????????? window of opportunity, limitations in bandwidth, and ?????????????? latency make this attack significantly more difficult ?????????????? to execute. ?????????????? RSA key exchange is enabled by default although least ?????????????? prioritized if server order is honored. For such a ?????????????? cipher suite to be chosen it must also be supported by ?????????????? the client and probably the only shared cipher suite. ?????????????? Captured TLS sessions encrypted with ephemeral cipher ?????????????? suites (DHE or ECDHE) are not at risk for subsequent ?????????????? decryption due to this vulnerability. ?????????????? As a workaround if default cipher suite configuration ?????????????? was used you can configure the server to not use ?????????????? vulnerable suites with the ciphers option like this: ?????????????? {ciphers, [Suite || Suite <- ssl:cipher_suites(), ?????????????? element(1,Suite) =/= rsa]} ?????????????? that is your code will look somethingh like this: ?????????????? ssl:listen(Port, [{ciphers, [Suite || Suite <- ?????????????? ssl:cipher_suites(), element(1,S) =/= rsa]} | ?????????????? Options]). ?????????????? Thanks to Hanno B?ck, Juraj Somorovsky and Craig Young ?????????????? for reporting this vulnerability. ?Full runtime dependencies of ssl-7.3.3.2: crypto-3.3, erts-7.0, ?inets-5.10.7, kernel-3.0, public_key-1.0, stdlib-2.0 ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- ?--------------------------------------------------------------------- From erlang@REDACTED Thu Nov 23 17:35:02 2017 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 23 Nov 2017 17:35:02 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511379790.16757707@apps.rackspace.com> References: <1511379790.16757707@apps.rackspace.com> Message-ID: I agree 100% If you make major changes to a library module that been around for a very long time you will break a lot of old code. At a first cast It's OK to add new functions to the module It's ok to fix bugs in old functions with changing the names But it NOT ok to remove functions or change the semantics of existing non-buggy functions. It's not as if we'll run out of module names soon. Call the new module strungs_v1 strings_1 or better_strings or strings_improved or anything you feel like but please^1000 don't break my old code. If I'm just reading code and I see strings:somefunc(...) then I'd very much like to know that there is only ONE version of the strings module. If two things have the same name they should be the same. One of the really really good things about Erlang is that I can take 20 year old code and recompile it and it just works with a very high probability. Please don't break my old code by changing the standard libraries. Just as an aside - in my hobby projects all my library code is in one directory (not multiple directories - this makes it easy to decide which directory to put things in) Libraries have names like mod.erl thereafter mod_vsn1.erl mod_vsn2.erl mod_vsn3.erl etc. and I don't make big changes old module once written. If by any mechanism whatsoever you can create a situation where two things with the same name are significantly different you'll end up with a large number of problems. Really modules should have no names but be named by (say) the SHA1 of their content and that way there would be no naming errors - but we don't really know how to do this in a convenient way yet. Cheers /Joe On Wed, Nov 22, 2017 at 8:43 PM, wrote: > Dear Gods of Erlang, > > > > "This module has been reworked in Erlang/OTP 20 to handle unicode:chardata() > and operate on grapheme clusters. The old functions that only work on > Latin-1 lists as input are still available but should not be used. They will > be deprecated in Erlang/OTP 21." > > > > I'm sorry. I've brought up this issue before and got lots of push back. > > > > But every time I look up tried and true and long-used string functions to > find that they are deprecated and will be dropped in future Erlang releases > my blood pressure soars. Both my wife and my doctor tell me that at my age > this is a dangerous thing. > > > > I do understand the importance and necessity of Unicode. And applaud the > addition of Unicode functions. > > > > But the deprecated string functions have a long history. The English > language and Latin-1 characters are widely used around the world. > > > > Yes, it should be easy for programmers to translate code from one user > language to another. But I'm not convinced that the Gods of Erlang have > found the optimal solution by dropping all Latin-1 string functions. > > > > My particular application is directed toward English speakers. So, until > further notice, I have no use for Unicode. > > > > I don't want to sound like nationalist pig, but I think dropping the Latin-1 > string functions from future Erlang releases is a BIG mistake. > > > > I look up tokens/2, a function that I use fairly frequently, and I see that > it's deprecated. I look up the suggested replacement and I see lexemes/2. > > > > So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I > see that a lexeme is "a meaningful linguistic unit." > > > > Meaning what? I just want to turn "this and that" into "This And That." > > > > I read further in the Erlang docs and I see "grapheme cluster." WHAT THE > ... IS GRAPHEME CLUSTER? > > > > I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a unit > of a writing system." > > > > Ah yes, grapheme is defined in the docs. But I have to read and re-read the > definition to understand what the God's of Erlang mean by a "graphene > cluster." And I'm still not sure I get it. > > > > It sounds like someone took a linguistics class and is trying to show off. > > > > But now I've spent 30 minutes--- time that I don't have to waste trying to > figure out how do a simple manipulation of "this and that." Recurse the next > time I want to look up a string function in the Erlang docs. > > > > SOLUTION > > > > Keep the Latin-1 string functions. Put them in a separate library if > necessary. Or put the new Unicode functions in a separate library. But don't > arbitrarily drop them. > > > > Some folks have suggested that I maintain my own library of the deprecated > Latin1 functions. But why should I have to do that? How does that help other > folks with the same issue? > > > > Bottom line: please please please do not drop the existing Latin-1 string > functions. > > > > Please don't. > > > > Best wishes, > > > > LRP > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From erlang@REDACTED Thu Nov 23 17:39:33 2017 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 23 Nov 2017 17:39:33 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <03e7d789-6487-b2dc-0a5b-698d97078ced@ninenines.eu> References: <1511379790.16757707@apps.rackspace.com> <03e7d789-6487-b2dc-0a5b-698d97078ced@ninenines.eu> Message-ID: On Wed, Nov 22, 2017 at 9:28 PM, Lo?c Hoguin wrote: > Calm down. Considering how ubiquitous the string module is, these functions > are not going to be removed for at least a few years. That gives you plenty > of time to understand the new string module. If you change it in a 1000 years time you're really going to confuse everybody. Programs in 3020 will want to know which millenia the code was written. There is no shortage of names. Call the new module string_vsn1 and NOT string then string and string_vsn1 can co-exist *forever* /Joe > > Perhaps during this journey you can help make the documentation for the > module more user friendly by sending patches or opening tickets at > bugs.erlang.org. I'll admit that the current documentation does confuse me > personally, though I've not needed to use it yet. > > Unfortunately languages are complex and Unicode is therefore also complex. > There's no real way around that. Even if you target English speakers it's > likely that you will need Unicode, because many things require it like names > or addresses for example. So even if it feels like you won't need it (and > maybe you won't) it's a good idea to be ready for it. > > I wouldn't say latin1 is widely used anymore. Most of everything uses > Unicode nowadays. Nearly everything switched to Unicode, Erlang is one of > the last. Even your email was sent encoded in utf-8. > > On 11/22/2017 08:43 PM, lloyd@REDACTED wrote: >> >> Dear Gods of Erlang, >> >> "This module has been reworked in Erlang/OTP 20 to handle >> unicode:chardata() >> and operate on grapheme clusters. The old functions >> that only work on Latin-1 >> lists as input are still available but should not be used. They will be >> deprecated in Erlang/OTP 21." >> >> I'm sorry. I've brought up this issue before and got lots of push back. >> >> But every time I look up tried and true and long-used string functions to >> find that they are deprecated and will be dropped in future Erlang releases >> my blood pressure soars. Both my wife and my doctor tell me that at my age >> this is a dangerous thing. >> >> I do understand the importance and necessity of Unicode. And applaud the >> addition of Unicode functions. >> >> But the deprecated string functions have a long history. The English >> language and Latin-1 characters are widely used around the world. >> >> Yes, it should be easy for programmers to translate code from one user >> language to another. But I'm not convinced that the Gods of Erlang have >> found the optimal solution by dropping all Latin-1 string functions. >> >> My particular application is directed toward English speakers. So, until >> further notice, I have no use for Unicode. >> >> I don't want to sound like nationalist pig, but I think dropping the >> Latin-1 string functions from future Erlang releases is a BIG mistake. >> >> I look up tokens/2, a function that I use fairly frequently, and I see >> that it's deprecated. I look up the suggested replacement and I see >> lexemes/2. >> >> So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I >> see that a lexeme is "a meaningful linguistic unit." >> >> Meaning what? I just want to turn "this and that" into "This And That." >> >> I read further in the Erlang docs and I see "grapheme cluster." WHAT THE >> ... IS GRAPHEME CLUSTER? >> >> I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a >> unit of a writing system." >> >> Ah yes, grapheme is defined in the docs. But I have to read and re-read >> the definition to understand what the God's of Erlang mean by a "graphene >> cluster." And I'm still not sure I get it. >> >> It sounds like someone took a linguistics class and is trying to show off. >> >> But now I've spent 30 minutes--- time that I don't have to waste trying to >> figure out how do a simple manipulation of "this and that." Recurse the next >> time I want to look up a string function in the Erlang docs. >> >> SOLUTION >> >> Keep the Latin-1 string functions. Put them in a separate library if >> necessary. Or put the new Unicode functions in a separate library. But don't >> arbitrarily drop them. >> >> Some folks have suggested that I maintain my own library of the deprecated >> Latin1 functions. But why should I have to do that? How does that help other >> folks with the same issue? >> >> Bottom line: please please please do not drop the existing Latin-1 string >> functions. >> >> Please don't. >> >> Best wishes, >> >> LRP >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -- > Lo?c Hoguin > https://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From michal@REDACTED Thu Nov 23 21:39:11 2017 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Thu, 23 Nov 2017 21:39:11 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <1511379790.16757707@apps.rackspace.com> Message-ID: On 23 Nov 2017, 17:35 +0100, Joe Armstrong , wrote: > I agree 100% > > If you make major changes to a library module that been around for a very long > time you will break a lot of old code. > > At a first cast > > It's OK to add new functions to the module > It's ok to fix bugs in old functions with changing the names > > But it NOT ok to remove functions or change the semantics of existing > non-buggy functions. > > It's not as if we'll run out of module names soon. Call the new module > > strungs_v1 strings_1 or better_strings or strings_improved or > anything you feel like but > While this sounds great, I will argue that it's not very practical. The primary problem is that somebody has now to maintain both versions of the code. And there are situation when even old code needs to change - the particular case we're probably all agree with is when security issues are discovered. If the team (which we assume is of fixed size) spends their time maintaining old code, they don't spend time developing new features. Resources are unfortunately limited. Another downside of keeping all old implementations is that it decreases readability of code. Code is read much more often than written and should be optimised for reading. But now, each time I see strings:tokens/1 and strings_v1:tokens/1 I need to decide if they actually do the same when reading the code. And I need to do this every time I read the code. The same distinction is needed during an upgrade, but it's needed only once. Micha?. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexakarpov@REDACTED Thu Nov 23 22:09:46 2017 From: alexakarpov@REDACTED (Alexandre Karpov) Date: Thu, 23 Nov 2017 16:09:46 -0500 Subject: [erlang-questions] [erlang-question][maps][comprehension] In-Reply-To: References: Message-ID: Found this year-old question while looking for any leads on the maps comprehension status.. LYSE does have a "PS" chapter, that seems to suggest the work wasn't finished yet at the time of writing (R17): http://learnyousomeerlang.com/maps#what-maps-shall-be ...and the EEP-43 linked is status 'draft'. The code in LYSE fails with a syntax error, too. Though that's not a map but a list comprehension, it seems to be related to using the map as a data source for the generator expression: 9> Weather = #{toronto => rain, montreal => storms, london => fog, paris => sun, boston => fog, vancouver => snow}. 10> FoggyPlaces = [X || X := fog <- Weather]. * 1: syntax error before: ':=' - the best I can infer from it is the usage of ":=" operator is illegal in this context (which would imply map comprehensions aren't supported). This wasn't entirely unexpected, I did run in something similar with maps examples presented in PE 2ed, which were supposed to work very soon, but still don't, and they also had to do with map de-structuring/pattern matching. (I ended up getting the expected result this way: 15> FoggyPlaces = [X || {X, Y} <- maps:to_list(Weather), Y == fog].) Creating a map with a comprehension isn't yet possible, too, it seems: 43> #{X => foggy || X <- [london,boston]}. * 1: syntax error before: '||' Again, this seems to suggest that "=>" isn't legal in this syntactic context. So, while the answer to the old question below remains correct - the person asking was trying to create a very esoteric map =) but even without that issue, things don't work as it was hinted they will in the near future. Anyone can provide any insight on the state of maps? The reason I am interested in getting to the bottom of this is, I want to know if the problem is with Erlang syntax, or is it due to some issues related to the runtime system itself; in the former case, I could explore Elixir or LFE, but in the latter, I just need to wait for EEP to move forward =) Thank you. On Wed, Jun 22, 2016 at 8:59 PM, Richard A. O'Keefe wrote: > > > On 23/06/16 7:22 AM, Sachin Panemangalore wrote: > >> Hi >> A question on comprehension in erlang in general. >> >> [{X,Y} || X<-[1,2,3], Y<-[a,b,c]]. works >> >> but >> >> #{X=>Y || X<-[1,2,3], Y<-[a,b,c] } does not >> >> any specific reason why this was not implemented ( since Erlang is >> relative mature my guess is its a design choice by the creators of Erlang ). >> >> What is it supposed to mean? Expanding the generators out, > #{ 1 => a, 1 => b, 1 => c, 2 => a, 2 => b, 2 => c, 3 => a, 3 => b, 3 => c } > is not really a meaningful map. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christopher.meiklejohn@REDACTED Thu Nov 23 22:54:09 2017 From: christopher.meiklejohn@REDACTED (Christopher Meiklejohn) Date: Thu, 23 Nov 2017 21:54:09 +0000 Subject: [erlang-questions] disksup crashing when spawned/forked from process Message-ID: I've got an Erlang/Elixir application that's being invoked via child_process in Node.js. Outside of Node.js, it's fine. Inside, the node crashes on startup with a badarg from disksup -- this is because the application requires that os_mon be started. A trivial reproducer is the following: var child_process = require('child_process'); child_process.spawn(path_to_app_executable, ['foreground'], { stdio: ignore, detached: true }); Included is the crash: 2017-11-23 21:26:04 =ERROR REPORT==== ** Generic server disksup terminating ** Last message in was timeout ** When Server state == [{data,[{"OS",{unix,linux}},{"Timeout",1800000},{"Threshold",80},{"D iskData",[]}]}] ** Reason for termination == ** {badarg,[{erlang,port_close,[#Port<0.2000>],[]},{disksup,terminate,2,[{file,"disksup.erl" },{line,169}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,629}]},{gen_server ,terminate,7,[{file,"gen_server.erl"},{line,795}]},{proc_lib,init_p_do_apply,3,[{file,"proc_ lib.erl"},{line,247}]}]} 2017-11-23 21:26:04 =CRASH REPORT==== crasher: initial call: disksup:init/1 pid: <0.913.0> registered_name: disksup exception exit: {{badarg,[{erlang,port_close,[#Port<0.2000>],[]},{disksup,terminate,2,[{ file,"disksup.erl"},{line,169}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line, 629}]},{gen_server,terminate,7,[{file,"gen_server.erl"},{line,795}]},{proc_lib,init_p_do_app ly,3,[{file,"proc_lib.erl"},{line,247}]}]},[{gen_server,terminate,7,[{file,"gen_server.erl"} ,{line,800}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} ancestors: [os_mon_sup,<0.911.0>] messages: [] links: [<0.912.0>] dictionary: [] trap_exit: true status: running heap_size: 610 stack_size: 27 reductions: 368 neighbours: 2017-11-23 21:26:04 =SUPERVISOR REPORT==== Supervisor: {local,os_mon_sup} Context: child_terminated Reason: {badarg,[{erlang,port_close,[#Port<0.2000>],[]},{disksup,terminate,2,[{file ,"disksup.erl"},{line,169}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,629} ]},{gen_server,terminate,7,[{file,"gen_server.erl"},{line,795}]},{proc_lib,init_p_do_apply,3 ,[{file,"proc_lib.erl"},{line,247}]}]} Offender: [{pid,<0.913.0>},{id,disksup},{mfargs,{disksup,start_link,[]}},{restart_typ e,permanent},{shutdown,2000},{child_type,worker}] -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Thu Nov 23 23:00:56 2017 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 23 Nov 2017 23:00:56 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <1511379790.16757707@apps.rackspace.com> Message-ID: On Thu, Nov 23, 2017 at 9:39 PM, Micha? Muska?a wrote: > > On 23 Nov 2017, 17:35 +0100, Joe Armstrong , wrote: > > I agree 100% > > If you make major changes to a library module that been around for a very > long > time you will break a lot of old code. > > At a first cast > > It's OK to add new functions to the module > It's ok to fix bugs in old functions with changing the names > > But it NOT ok to remove functions or change the semantics of existing > non-buggy functions. > > It's not as if we'll run out of module names soon. Call the new module > > strungs_v1 strings_1 or better_strings or strings_improved or > anything you feel like but > > > While this sounds great, I will argue that it's not very practical. The > primary problem is that somebody has now to maintain both versions of the > code. Why maintain the old version? As soon as there is a strings_v1 exist then changes can be made there > And there are situation when even old code needs to change - the > particular case we're probably all agree with is when security issues are > discovered. Possibly - you could alwys issue a very strong compiler warning "security problem in strings" please change to strings_v1 > If the team (which we assume is of fixed size) spends their time > maintaining old code, they don't spend time developing new features. > Resources are unfortunately limited. > > Another downside of keeping all old implementations is that it decreases > readability of code. Code is read much more often than written and should be > optimised for reading. It should be but isn't - look at the history of erl_scan the first versions were very readable - later versions were heavily optimised and far less readable > But now, each time I see strings:tokens/1 and > strings_v1:tokens/1 I need to decide if they actually do the same when > reading the code. You could always write in strings_v1.erl tokens(X, Y) -> strings:tokens(X, Y). and it would be abundantly clear - or use a parse transform. > And I need to do this every time I read the code. The same > distinction is needed during an upgrade, but it's needed only once. I still maintain that things with the same name must be the same. As soon as you get two versions of strings offering different functions then the name of the module 'strings.erl' becomes ambiguous. You have to say "I mean the version of strings in version 19.2 of Erlang" Oh dear I thought you meant version 45.3" If we use a name we should not have to qualify it by either the date when the name was valid or by the checksum of the Git commit in which it can be found. Imagine what would happen if I could change my name on an arbitrary date I was talking to joe the other day, when? 12 June 2015 - Oh you mean when he was called fred - No that was later he changed his name to Donald on the 23 th August 2016. And what, is wrong with names like srings_vsn1, string_vsn_2 etc. it's not as if the integers are going to run out. One thing I've always hated about revision control systems like GIT is that the same name means different things in different commits. This causes no end of confusion and many errors. Breaking peoples code by changing changing the libraries I view as a fundamental sin. After a few iterations you'll end up with two mutually incompatible versions of a library with the same name. One will export a with you want to use, the other b which you also want to use. But you cannot use both. I have seen this in virtually every system I've every programmed. Just invent a new name if you change things. How difficult can that be? /Joe > > Micha?. From luke@REDACTED Thu Nov 23 23:21:55 2017 From: luke@REDACTED (Luke Bakken) Date: Thu, 23 Nov 2017 14:21:55 -0800 Subject: [erlang-questions] disksup crashing when spawned/forked from process In-Reply-To: References: Message-ID: Hi Chris - What is "path_to_app_executable"? erl itself or a relx-type shell script sort of thing? On Nov 23, 2017 1:54 PM, "Christopher Meiklejohn" < christopher.meiklejohn@REDACTED> wrote: I've got an Erlang/Elixir application that's being invoked via child_process in Node.js. Outside of Node.js, it's fine. Inside, the node crashes on startup with a badarg from disksup -- this is because the application requires that os_mon be started. A trivial reproducer is the following: var child_process = require('child_process'); child_process.spawn(path_to_app_executable, ['foreground'], { stdio: ignore, detached: true }); Included is the crash: 2017-11-23 21:26:04 =ERROR REPORT==== ** Generic server disksup terminating ** Last message in was timeout ** When Server state == [{data,[{"OS",{unix,linux}},{"Timeout",1800000},{"Threshold",80},{"D iskData",[]}]}] ** Reason for termination == ** {badarg,[{erlang,port_close,[#Port<0.2000>],[]},{disksup,terminate,2,[{file,"disksup.erl" },{line,169}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,629}]},{gen_server ,terminate,7,[{file,"gen_server.erl"},{line,795}]},{proc_lib,init_p_do_apply,3,[{file,"proc_ lib.erl"},{line,247}]}]} 2017-11-23 21:26:04 =CRASH REPORT==== crasher: initial call: disksup:init/1 pid: <0.913.0> registered_name: disksup exception exit: {{badarg,[{erlang,port_close,[#Port<0.2000>],[]},{disksup,terminate,2,[{ file,"disksup.erl"},{line,169}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line, 629}]},{gen_server,terminate,7,[{file,"gen_server.erl"},{line,795}]},{proc_lib,init_p_do_app ly,3,[{file,"proc_lib.erl"},{line,247}]}]},[{gen_server,terminate,7,[{file,"gen_server.erl"} ,{line,800}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} ancestors: [os_mon_sup,<0.911.0>] messages: [] links: [<0.912.0>] dictionary: [] trap_exit: true status: running heap_size: 610 stack_size: 27 reductions: 368 neighbours: 2017-11-23 21:26:04 =SUPERVISOR REPORT==== Supervisor: {local,os_mon_sup} Context: child_terminated Reason: {badarg,[{erlang,port_close,[#Port<0.2000>],[]},{disksup,terminate,2,[{file ,"disksup.erl"},{line,169}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,629} ]},{gen_server,terminate,7,[{file,"gen_server.erl"},{line,795}]},{proc_lib,init_p_do_apply,3 ,[{file,"proc_lib.erl"},{line,247}]}]} Offender: [{pid,<0.913.0>},{id,disksup},{mfargs,{disksup,start_link,[]}},{restart_typ e,permanent},{shutdown,2000},{child_type,worker}] _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Thu Nov 23 23:35:34 2017 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Thu, 23 Nov 2017 23:35:34 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: From Michael.K.Schmidt@REDACTED Thu Nov 23 23:46:38 2017 From: Michael.K.Schmidt@REDACTED (Michael Schmidt) Date: Thu, 23 Nov 2017 22:46:38 +0000 Subject: [erlang-questions] Mnesia deleting log file In-Reply-To: <394a0fc5-9857-14ef-f032-f5267dfe4627@utl.in> References: <7d0f3489-e7aa-e1c1-1222-ab2931ef815b@utl.in> <394a0fc5-9857-14ef-f032-f5267dfe4627@utl.in> Message-ID: Can you provide more details? Does this happen if you kill the VM? Or are you pulling power on the machine? If it?s the later case, what filesystem are you using? Also, please try setting dump_log_time_threshold to 500 or so. This will flush it within a half second. I don?t think you want dump_log_write_threshold set to 1?this will cause the file to be repeatedly written. The goal is to cache the writes in RAM so that a single write to the FS happens. From: Arun [mailto:arunp@REDACTED] Sent: Thursday, November 23, 2017 5:22 AM To: Michael Schmidt ; Jesper Louis Andersen Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Mnesia deleting log file Even after setting the "dump_log_update_in_place" configuration to false, the data is not getting retained after a power outage. Any more configurations that can be tried. Regards, Arun P On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote: I agree?setting dump_log_in_place to false makes extremely unlikely for mnesia to fail in this way. Both JFFS2 and UBIFS ensure that rename is atomic, even when power is pulled. For desktop applications, the culprit is likely the VM being killed, not the whole system crashing. Thus the rename was either executed or not; either way we have a complete DAT file. There is always a chance of things like this happening: http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but this shouldn?t happen on most hardware. Note that the default for dump_log_time_threshold is 3 minutes. Speed vs Safety is always a trade-off, but many people will be surprised to lose data in a crash 2 minutes after saving. From: Jesper Louis Andersen [mailto:jesper.louis.andersen@REDACTED] Sent: Wednesday, November 22, 2017 7:54 AM To: Michael Schmidt Cc: Arun ; erlang-questions@REDACTED Subject: Re: [erlang-questions] Mnesia deleting log file This thread makes we wonder about the following section in the Mnesia documentation: "If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority." In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should. With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question. [1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole. On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt > wrote: Hi Arun, If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data). We fight this issue on embedded devices a lot. Look ?Configuration Parameters? section here: http://erlang.org/doc/man/mnesia.html#id69584 You will want to set dump_log_update_in_place to false. This will make it write out the table to a new file, and rename it the very end. One other option to look at is dump_log_time_threshold to change how often items are flushed to disk. For configuration data, you can likely set this to a few seconds. These options can be set via the sys.config file as well Mike From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Arun Sent: Tuesday, November 21, 2017 10:16 PM To: erlang-questions@REDACTED Subject: [erlang-questions] Mnesia deleting log file Dear all, I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing all the persistent configurations. I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem and how do I fix it? Mnesia('test@REDACTED'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running, 'test@REDACTED'} ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________ _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Fri Nov 24 10:05:05 2017 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Fri, 24 Nov 2017 10:05:05 +0100 Subject: [erlang-questions] EEP 47: Syntax in try/catch to retrieve the stacktrace directly Message-ID: There is a new EEP that proposes a new syntax in try/catch to retrieve the stacktrace directly without using erlang:get_stacktrace/0: https://github.com/erlang/eep/blob/master/eeps/eep-0047.md /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From eric.des.courtis@REDACTED Fri Nov 24 14:13:49 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Fri, 24 Nov 2017 08:13:49 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: Message-ID: While I feel the pain of Joe having suffered countless hours fixing deprecation in various old libraries. I have to say there is something to be said for throwing things away. I can't help but think I can have my cake and eat it too. Maybe something like a transpiler to compile old Erlang to new Erland and keep the Erlang VM lean? On Thu, Nov 23, 2017 at 5:35 PM, ?ric Pailleau wrote: > Hi, > > If Erlang had namespaces we could imagine old strings function if no > namespace, for old code compat, while new code could use same functions > names, but with a namespace. Let say -namespace(strings, pref1, " > http://www.erlang.org/strings/utf8"). In header. > > When using functions with mixed namespaces, Use of a prefix would solve > the problem. > > pref1::strings:func() for new utf8 module or strings:func() for legacy > latin1 ones. > > This would also solve the very complicated problem of module name clashes > from different repositories... > > Regards > > > Le 23 nov. 2017 11:00 PM, Joe Armstrong a ?crit : > > On Thu, Nov 23, 2017 at 9:39 PM, Micha? Muska?a wrote: > > > > On 23 Nov 2017, 17:35 +0100, Joe Armstrong , wrote: > > > > I agree 100% > > > > If you make major changes to a library module that been around for a very > > long > > time you will break a lot of old code. > > > > At a first cast > > > > It's OK to add new functions to the module > > It's ok to fix bugs in old functions with changing the names > > > > But it NOT ok to remove functions or change the semantics of existing > > non-buggy functions. > > > > It's not as if we'll run out of module names soon. Call the new module > > > > strungs_v1 strings_1 or better_strings or strings_improved or > > anything you feel like but > > > > > > While this sounds great, I will argue that it's not very practical. The > > primary problem is that somebody has now to maintain both versions of the > > code. > > Why maintain the old version? As soon as there is a strings_v1 exist > then changes can be made there > > > And there are situation when even old code needs to change - the > > particular case we're probably all agree with is when security issues are > > discovered. > > Possibly - you could alwys issue a very strong compiler warning > "security problem in strings" please change to strings_v1 > > > If the team (which we assume is of fixed size) spends their time > > maintaining old code, they don't spend time developing new features. > > Resources are unfortunately limited. > > > > Another downside of keeping all old implementations is that it decreases > > readability of code. Code is read much more often than written and > should be > > optimised for reading. > > It should be but isn't - look at the history of erl_scan the first > versions were > very readable - later versions were heavily optimised and far less readable > > > But now, each time I see strings:tokens/1 and > > strings_v1:tokens/1 I need to decide if they actually do the same when > > reading the code. > > You could always write in strings_v1.erl > > tokens(X, Y) -> > strings:tokens(X, Y). > > and it would be abundantly clear - or use a parse transform. > > > And I need to do this every time I read the code. The same > > distinction is needed during an upgrade, but it's needed only once. > > I still maintain that things with the same name must be the same. > As soon as you get two versions of strings offering different > functions then the name of the module 'strings.erl' becomes > ambiguous. > > You have to say "I mean the version of strings in version 19.2 of Erlang" > Oh dear I thought you meant version 45.3" > > If we use a name we should not have to qualify it by either the date > when the name was valid or by the checksum of the Git commit in which > it can be found. > > Imagine what would happen if I could change my name on an arbitrary date > > I was talking to joe the other day, when? 12 June 2015 - Oh you mean > when he was called > fred - No that was later he changed his name to Donald on the 23 th August > 2016. > > And what, is wrong with names like srings_vsn1, string_vsn_2 etc. it's > not as if the > integers are going to run out. > > One thing I've always hated about revision control systems like GIT is > that the same name > means different things in different commits. This causes no end of > confusion and many errors. > > Breaking peoples code by changing changing the libraries I view as a > fundamental sin. > > After a few iterations you'll end up with two mutually incompatible > versions of a library > with the same name. One will export a with you want to use, the other > b which you also want to use. > But you cannot use both. > > I have seen this in virtually every system I've every programmed. > > Just invent a new name if you change things. > > How difficult can that be? > > /Joe > > > > > Micha?. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Fri Nov 24 14:37:11 2017 From: zxq9@REDACTED (zxq9) Date: Fri, 24 Nov 2017 22:37:11 +0900 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: Message-ID: <7326453.QddQBcBGA1@changa> On 2017?11?24? ??? 08:13:49 Eric des Courtis wrote: > While I feel the pain of Joe having suffered countless hours fixing > deprecation in various old libraries. I have to say there is something to > be said for throwing things away. > > I can't help but think I can have my cake and eat it too. Maybe something > like a transpiler to compile old Erlang to new Erland and keep the Erlang > VM lean? If the new functions did what the old functions do, then sure. But that's not how thing have turned out. On reflection, I actually think the new string functions should have been rolled into a "utf8" module. Or something. And the "strings" module could either have had implementation adjustments that use the utf8 utilities underneath or been left alone to deal with latin1 (but either way be amply documented). I'm the heaviest unicode string dealing guy I know. I'm SUPER happy that the idea of "string" has been advanced (finally!) to mean "unicode strings". But breakage is a thing, and Lloyd and Joe have a point. -Craig From essen@REDACTED Fri Nov 24 14:59:52 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Fri, 24 Nov 2017 14:59:52 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <7326453.QddQBcBGA1@changa> References: <7326453.QddQBcBGA1@changa> Message-ID: <266acbfa-c8cd-a2c8-125f-9a168c6d2442@ninenines.eu> On 11/24/2017 02:37 PM, zxq9 wrote: > On 2017?11?24? ??? 08:13:49 Eric des Courtis wrote: >> While I feel the pain of Joe having suffered countless hours fixing >> deprecation in various old libraries. I have to say there is something to >> be said for throwing things away. >> >> I can't help but think I can have my cake and eat it too. Maybe something >> like a transpiler to compile old Erlang to new Erland and keep the Erlang >> VM lean? > > If the new functions did what the old functions do, then sure. > > But that's not how thing have turned out. > > On reflection, I actually think the new string functions should have > been rolled into a "utf8" module. Or something. And the "strings" module > could either have had implementation adjustments that use the utf8 > utilities underneath or been left alone to deal with latin1 (but either > way be amply documented). > > I'm the heaviest unicode string dealing guy I know. I'm SUPER happy that > the idea of "string" has been advanced (finally!) to mean "unicode strings". > But breakage is a thing, and Lloyd and Joe have a point. Nothing has been broken as far as I understand. I'm not sure what's the big deal. New set of functions has been added, old set of functions still works the same, still documented but will very slowly be rolled out. Joe's ideals are very much unrealistic, it's very unlikely that computers work the same in 1000 years, it's very unlikely that Erlang is still around then, that anyone remembers the programs we wrote today or heck even that our civilizations still exist. The odds are not much greater for 100 years in the future as far as the code goes. The problem is much more practical, and one practical aspect to think about is maintenance. Old code has to be removed eventually to lower the maintenance costs (for OTP team). Still, I doubt this particular code will be removed in the next 10 years. I'd bet something like gen_fsm would be removed far quicker than the old string functions. Lloyd's issues seem to be more about documentation, specifically it lacking examples for converting to the new functions. This kind of issue will be resolved long before the old functions are removed. OTP has slowly phased out a lot of code. It's nothing new. This time it's both a combination of this and the world phasing out latin1, so that's more reasons to move forward. -- Lo?c Hoguin https://ninenines.eu From eric.des.courtis@REDACTED Fri Nov 24 15:07:11 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Fri, 24 Nov 2017 09:07:11 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <7326453.QddQBcBGA1@changa> References: <7326453.QddQBcBGA1@changa> Message-ID: >From my perspective, we want the following things: - The libraries to stay simple, small and clean (that means throwing away things) - Our old code to continue to work without modifications (that means having a mechanism for compiling old code targetting new versions of Erlang) - To mix old and new Erlang code together I see no reason why we can't have both. What I am not okay with is: - Endlessly growing libraries of functions that are there for legacy reasons (wasting space and confusing new developers) - Fixing libraries over and over again because something got marked deprecated So how do we get both? On Fri, Nov 24, 2017 at 8:37 AM, zxq9 wrote: > On 2017?11?24? ??? 08:13:49 Eric des Courtis wrote: > > While I feel the pain of Joe having suffered countless hours fixing > > deprecation in various old libraries. I have to say there is something to > > be said for throwing things away. > > > > I can't help but think I can have my cake and eat it too. Maybe something > > like a transpiler to compile old Erlang to new Erland and keep the Erlang > > VM lean? > > If the new functions did what the old functions do, then sure. > > But that's not how thing have turned out. > > On reflection, I actually think the new string functions should have > been rolled into a "utf8" module. Or something. And the "strings" module > could either have had implementation adjustments that use the utf8 > utilities underneath or been left alone to deal with latin1 (but either > way be amply documented). > > I'm the heaviest unicode string dealing guy I know. I'm SUPER happy that > the idea of "string" has been advanced (finally!) to mean "unicode > strings". > But breakage is a thing, and Lloyd and Joe have a point. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Fri Nov 24 15:22:51 2017 From: zxq9@REDACTED (zxq9) Date: Fri, 24 Nov 2017 23:22:51 +0900 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> Message-ID: <3254145.3rMIGXJoQI@changa> On 2017?11?24? ??? 09:07:11 you wrote: > So how do we get both? The eternal million-dollar-question. My (probably crappy) solution is the same for this as for untangling dependency knots: semver + static dependency declarations This also means no automatic dependency upgrades. I'm not sure how to make this a part of Erlang itself, though, so for now my way of doing things involves making it possible to execute different ERTS versions and making each execution of a program determine from its metadata what specific libs to pull, build and include in the execution environment (that is, do not allow "latest available", instead be explicit). ...which means, of course an external tool that does all that, and a repo that knows what you are asking for. It is up to a packager to define explicit dependencies (but dependencies are flat declarations, not trees that might conflict -- working that out is the packager's main burden, and this means you get bug-reproducible re-runs within a specific version of a package). I've never attempted to make a cross-platform utility that could install various versions of Erlang on the fly (Windows is always a mystery) but a cheap way of leaning on kerl has worked for me in the past on Linux. But again, it requires extra declarations on the part of a packager, which means someone has to test on various versions. This rarely happens, so I cheat and tend to wind up always just writing against the latest ERTS release -- and then we run into the core lib changes problem again. Anyway, its a circle of fun. Runtime and stdlib changes are always white hot with friction. -Craig From essen@REDACTED Fri Nov 24 15:40:40 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Fri, 24 Nov 2017 15:40:40 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> Message-ID: <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> On 11/24/2017 03:07 PM, Eric des Courtis wrote: > From my perspective, we want the following things: > > * The libraries to stay simple, small and clean (that means throwing > away things) > * Our old code to continue to work without modifications (that means > having a mechanism for compiling old code targetting new versions of > Erlang) That can work in many cases but not in the general case. This only works for a specific set of changes where an equivalent still exists in the code base. For example removing ciphers or hashes in crypto requires human intervention on every code bases that require them. Another issue comes from old code requiring syntax or VM features that have been removed. Then you have issues where it is technically still possible to do the old functionality, but the performance or memory aspects are not equivalent. > * To mix old and new Erlang code together If it's maintained, no problem. If not, even if it compiles and runs you should probably make sure to have a good set of tests to know it still works as intended. You'll want to do this even if OTP team go out of their way to make sure everything stays compatible forever. > I see no reason why we can't have both. What I am not okay with is: > > * Endlessly growing libraries of functions that are there for legacy > reasons (wasting space and confusing new developers) > * Fixing libraries over and over again because something got marked > deprecated > > So how do we get both? You don't. In fact your second point is incorrect: there's absolutely no reason to fix a library because something got marked deprecated. Zero. Developers should not "fix" libraries when something gets marked deprecated. Developers should *take notice* that something has been deprecated. There's nothing to fix to begin with! Check your OCD levels and stop trying to get rid of all the deprecation warnings with obscure parse transforms or version-specific defines... The time to fix comes when the feature gets *removed* or is rendered *incompatible*. (Or soon before that actually happens, anyway.) As far as the string module is concerned, there's absolutely no need to spend efforts to make your code run for both the old and the new string module because the functions will not be removed for a very long time. And until they are, chances are you will stop supporting OTP versions before the new functions were introduced. You can do the changes THEN. I am baffled every time someone opens a ticket telling me that I have deprecation warnings. I know! But they're harmless and the code works on all supported versions, so why would I spend efforts to remove it? Your question can be reframed to be about an endlessly growing library + having to fix libraries when things get removed, which is more of a choice, you can't really have both. Either you have an endlessly growing library, or you remove functions and fix libraries (manually or otherwise). -- Lo?c Hoguin https://ninenines.eu From silas@REDACTED Fri Nov 24 15:50:38 2017 From: silas@REDACTED (Silas) Date: Fri, 24 Nov 2017 12:50:38 -0200 Subject: [erlang-questions] Wrap my mind around NoSQL (Mnesia) Message-ID: <20171124145038.GA23275@auron> Hi! (This is mainly a question about NoSQL conceptions. If it is the wrong place to ask, sorry for that and, please, let me know). After years using RDBMS (with structured and OO paradigms at the programming language side) I'm diving into functional programming. When choosing a programming language to practice, I immediately fell in love with Erlang for this simplicity and beauty. After reading some documentation, some chapters of a book and trying simple projects I decided to try a more complex project with Mnesia. It is a social bookmarks website (something similar to what del.icio.us is) where user can bookmark a URL and associate one or more tags with it. For instance, some could bookmark "erlang.org" to tags "functionalprogramming", "programminglanguage" and others. Users also can query other users bookmarks, see what links are associated with a tag and what tags are associated with a given bookmark. In my mind, an RDBMS is a perfect solution for this problem. I'm trying to fit it into the key -> value model, though. My first question is: is this possible or I'm just doing the wrong thing by forcing the wrong solution? For what I understood, I need to get rid of normalization rules. So I devised something more or less like that: User table: {username, {urls, [tags]}} % set UserTag table: {user, [tags]} % set TagUrls table: {tag, url} % bag TagUsers table: {tag, user} % bag Urlusers table: {url, user} %bag UrlTags table: {url, tag} %bag It is still a bit confusing where I'd use a set, a bag or an ordered_set but I think you got the whole picture. Any help? Thanks! -- Silas From erlang@REDACTED Fri Nov 24 16:21:09 2017 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 24 Nov 2017 16:21:09 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> Message-ID: On Fri, Nov 24, 2017 at 3:40 PM, Lo?c Hoguin wrote: > On 11/24/2017 03:07 PM, Eric des Courtis wrote: >> >> From my perspective, we want the following things: >> >> * The libraries to stay simple, small and clean (that means throwing >> away things) >> * Our old code to continue to work without modifications (that means >> having a mechanism for compiling old code targetting new versions of >> Erlang) > > > That can work in many cases but not in the general case. This only works for > a specific set of changes where an equivalent still exists in the code base. > > For example removing ciphers or hashes in crypto requires human intervention > on every code bases that require them. > > Another issue comes from old code requiring syntax or VM features that have > been removed. > > Then you have issues where it is technically still possible to do the old > functionality, but the performance or memory aspects are not equivalent. > >> * To mix old and new Erlang code together > > > If it's maintained, no problem. If not, even if it compiles and runs you > should probably make sure to have a good set of tests to know it still works > as intended. > > You'll want to do this even if OTP team go out of their way to make sure > everything stays compatible forever. > >> I see no reason why we can't have both. What I am not okay with is: >> >> * Endlessly growing libraries of functions that are there for legacy >> reasons (wasting space and confusing new developers) >> * Fixing libraries over and over again because something got marked >> deprecated >> >> So how do we get both? > > > You don't. > > In fact your second point is incorrect: there's absolutely no reason to fix > a library because something got marked deprecated. > > Zero. > > Developers should not "fix" libraries when something gets marked deprecated. > Developers should *take notice* that something has been deprecated. > > There's nothing to fix to begin with! Check your OCD levels and stop trying > to get rid of all the deprecation warnings with obscure parse transforms or > version-specific defines... > > The time to fix comes when the feature gets *removed* or is rendered > *incompatible*. (Or soon before that actually happens, anyway.) > > As far as the string module is concerned, there's absolutely no need to > spend efforts to make your code run for both the old and the new string > module because the functions will not be removed for a very long time. And > until they are, chances are you will stop supporting OTP versions before the > new functions were introduced. You can do the changes THEN. > > I am baffled every time someone opens a ticket telling me that I have > deprecation warnings. I know! But they're harmless and the code works on all > supported versions, so why would I spend efforts to remove it? > > Your question can be reframed to be about an endlessly growing library + > having to fix libraries when things get removed, which is more of a choice, > you can't really have both. Either you have an endlessly growing library, or > you remove functions and fix libraries (manually or otherwise). You are quite right - what I'd like is a garbage collector for code. Given a root query and a set of modules remove all code that can never be reached. Actually the way Knuth did things was pretty good TeX provides a great example. The version numbers were the digits of Pi TeX is currently at version 3.14159265 Knuth offered a reward of $2.56 for the first error in TeX to be doubled for each new error reported. TeX was implemented as an initial program, followed by a set of patch files) to get to version you take the first version then apply a set of patches to get to the version you want. We could do this in erlang - some syntax like: -module(string) -vsn(10) -remove([func1/4, func4/3]). func7() -> .... etc. This might create a module called string_10.erl and would be a copy of all the code in string_9.erl the remove declaration just removes code from string_9.erl Any additional functions in the module are just added to the new module. This way the old and new code could happily co-exist. For developers there could be a "window" of code they might maintain say from string_7 to string_10 and this would migrate forwards with time, This scheme worked well for Knuth but assumes a single thread of development with no branches. Cheers /Joe > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From cean.ebengt@REDACTED Fri Nov 24 16:29:29 2017 From: cean.ebengt@REDACTED (bengt e) Date: Fri, 24 Nov 2017 16:29:29 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> Message-ID: Greetings, Would something like 'go fix' be possible for Erlang, in this particular case? bengt On Fri, Nov 24, 2017 at 3:07 PM, Eric des Courtis < eric.des.courtis@REDACTED> wrote: > From my perspective, we want the following things: > > - The libraries to stay simple, small and clean (that means throwing > away things) > - Our old code to continue to work without modifications (that means > having a mechanism for compiling old code targetting new versions of Erlang) > - To mix old and new Erlang code together > > I see no reason why we can't have both. What I am not okay with is: > > - Endlessly growing libraries of functions that are there for legacy > reasons (wasting space and confusing new developers) > - Fixing libraries over and over again because something got marked > deprecated > > So how do we get both? > > On Fri, Nov 24, 2017 at 8:37 AM, zxq9 wrote: > >> On 2017?11?24? ??? 08:13:49 Eric des Courtis wrote: >> > While I feel the pain of Joe having suffered countless hours fixing >> > deprecation in various old libraries. I have to say there is something >> to >> > be said for throwing things away. >> > >> > I can't help but think I can have my cake and eat it too. Maybe >> something >> > like a transpiler to compile old Erlang to new Erland and keep the >> Erlang >> > VM lean? >> >> If the new functions did what the old functions do, then sure. >> >> But that's not how thing have turned out. >> >> On reflection, I actually think the new string functions should have >> been rolled into a "utf8" module. Or something. And the "strings" module >> could either have had implementation adjustments that use the utf8 >> utilities underneath or been left alone to deal with latin1 (but either >> way be amply documented). >> >> I'm the heaviest unicode string dealing guy I know. I'm SUPER happy that >> the idea of "string" has been advanced (finally!) to mean "unicode >> strings". >> But breakage is a thing, and Lloyd and Joe have a point. >> >> -Craig >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri Nov 24 16:32:10 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 24 Nov 2017 10:32:10 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> Message-ID: <20171124153209.GE2024@ferdmbp.local> On 11/24, bengt e wrote: >Greetings, > >Would something like 'go fix' be possible for Erlang, in this particular >case? > Not really. Functions like string:chr return a position of a matching character. There is no telling what that position will be used for later on. Another example is the ssl_verify_fun library. That one does hostname validation for SSL certificates. Replacing string functions working on ASCII/latin-1/ISO-8859-1 by unicode-aware strings that do normalization and whatnot is actually a potential security risk. Those are not entirely equivalent problem domains, and not all strings are equivalent in all contexts and encodings. From mononcqc@REDACTED Fri Nov 24 16:50:20 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 24 Nov 2017 10:50:20 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <7326453.QddQBcBGA1@changa> References: <7326453.QddQBcBGA1@changa> Message-ID: <20171124155019.GF2024@ferdmbp.local> On 11/24, zxq9 wrote: >On 2017?11?24? ??? 08:13:49 Eric des Courtis wrote: >On reflection, I actually think the new string functions should have >been rolled into a "utf8" module. Or something. And the "strings" module >could either have had implementation adjustments that use the utf8 >utilities underneath or been left alone to deal with latin1 (but either >way be amply documented). > This would be bad naming; utf8 is but one encoding of Unicode as I'm sure you're aware. The current string module should be able to handle utf8, utf16, utf32, *and* lists of codepoints (chardata()). 'strings' is sadly the most appropriate name for this module, and for the first time it also has the ability to really handle *anything* we consider to be a string: lists, binaries, and mixes of both. The problem was really that the old 'string' module was not super great at being a 'string' module. It would have been better named as 'cstring' or something. Maybe it could have been 'str' instead, who knows. Then we'd get cool conference talks saying how shitty and confusing the stdlib is because you don't know whether to use string or str in your code! Please let's not get inspired by 'mysql_real_escape_string', which had to be implemented because 'mysql_escape_string' was not good enough but people kept relying on it. It's a laughing stock and even a security problem for everyone in that case. >I'm the heaviest unicode string dealing guy I know. I'm SUPER happy that >the idea of "string" has been advanced (finally!) to mean "unicode strings". >But breakage is a thing, and Lloyd and Joe have a point. > I think we have to consider a thing, as someone else has mentioned: old code is not broken. Old code keeps running. Old code is fine. It has a new compiler warning. Old code could be wrong, because it may be getting unicode data and mangling it instead of doing the right thing. Old code could not even detect that. Old code was getting passed by everyone in the world. Old code can't even work safely on Erlang modules or the content of your .app file anymore because that content is now UTF-8. We can't know. For all we care, old code is getting to break because the world is passing it by and it's not keeping up. If old code must remain stable in a changing world, old code must be run in its old context: maybe stick it into the same old VM, or vendor it in along with its build tools, artifacts, and everything it needs (be careful though, R15 old code no longer can fetch packages or dependencies safely, since TLS from R15 is no longer safe in the real world). Old code can't necessarily be recompiled onto new tools, because new tools have to address a changing world. Sad thing for old code, but if you use old tools, maybe it will work. Maybe one can just copy/paste the string.erl module into cstring.erl, change the `-module` attribute, and then do a 'sed' call on their code. If old code is expected to work together with the existing API and no changes, but that the API is no longer right for the current world, maybe old code should freeze its dependencies and environment. In any case, current code is not yet old code. It's being told that eventually it will be old code. In the meanwhile, people who need to worry are those who need to support more than one version at once. Those people feel some pain for sure. The other people to suffer are those on a tight budget with humongous codebase whose life may be to keep maintaining changing code all the time because there's so much code they can hardly keep up with all the changes. So uh, what are you gonna promise in terms of breaking? It sounds like the best policy is to pick a value of what you promise in terms of backwards compatibility, and the OTP team has a very clear policy there. Two major versions at least. The risk of never changing nor breaking a thing forever is that old code can prevent new code from being written if old code is so bad that new code can't make sense. Then you have nothing but old code. You've accrued enough technical debt that nobody who hasn't grown with the system can now understand it. Then your old code for your old system is only possible to be used and maintain by old programmers, because the new programmers have gone somewhere else. From jamhedd@REDACTED Fri Nov 24 17:00:10 2017 From: jamhedd@REDACTED (Roman Galeev) Date: Fri, 24 Nov 2017 17:00:10 +0100 Subject: [erlang-questions] Wrap my mind around NoSQL (Mnesia) In-Reply-To: <20171124145038.GA23275@auron> References: <20171124145038.GA23275@auron> Message-ID: You can think about Mnesia as a multi-table key-value store, with some features added like transactions. And you can use Erlang terms as keys (say, a tuple). So, just tailor lookups to your needs, and it would be clear do you need a set or a bag. On Fri, Nov 24, 2017 at 3:50 PM, Silas wrote: > Hi! > > (This is mainly a question about NoSQL conceptions. If it is the wrong > place to ask, sorry for that and, please, let me know). > > After years using RDBMS (with structured and OO paradigms at the > programming language side) I'm diving into functional programming. When > choosing a programming language to practice, I immediately fell in love > with Erlang for this simplicity and beauty. > > After reading some documentation, some chapters of a book and trying > simple projects I decided to try a more complex project with Mnesia. It is > a social bookmarks website (something similar to what del.icio.us is) > where user can bookmark a URL and associate one or more tags with it. For > instance, some could bookmark "erlang.org" to tags > "functionalprogramming", "programminglanguage" and others. Users also can > query other users bookmarks, see what links are associated with a tag and > what tags are associated with a given bookmark. > > In my mind, an RDBMS is a perfect solution for this problem. I'm trying > to fit it into the key -> value model, though. My first question is: is > this possible or I'm just doing the wrong thing by forcing the wrong > solution? > > For what I understood, I need to get rid of normalization rules. So I > devised something more or less like that: > > User table: {username, {urls, [tags]}} % set > UserTag table: {user, [tags]} % set > TagUrls table: {tag, url} % bag > TagUsers table: {tag, user} % bag > Urlusers table: {url, user} %bag > UrlTags table: {url, tag} %bag > > It is still a bit confusing where I'd use a set, a bag or an ordered_set > but I think you got the whole picture. > > Any help? > > Thanks! > > -- > Silas > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- With best regards, Roman Galeev, +420 702 817 968 -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.des.courtis@REDACTED Fri Nov 24 17:26:58 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Fri, 24 Nov 2017 11:26:58 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> Message-ID: On Fri, Nov 24, 2017 at 9:40 AM, Lo?c Hoguin wrote: > On 11/24/2017 03:07 PM, Eric des Courtis wrote: > >> From my perspective, we want the following things: >> >> * The libraries to stay simple, small and clean (that means throwing >> away things) >> * Our old code to continue to work without modifications (that means >> having a mechanism for compiling old code targetting new versions of >> Erlang) >> > > That can work in many cases but not in the general case. This only works > for a specific set of changes where an equivalent still exists in the code > base. > > For example removing ciphers or hashes in crypto requires human > intervention on every code bases that require them. > > Another issue comes from old code requiring syntax or VM features that > have been removed. > > Then you have issues where it is technically still possible to do the old > functionality, but the performance or memory aspects are not equivalent. > > * To mix old and new Erlang code together >> > > If it's maintained, no problem. If not, even if it compiles and runs you > should probably make sure to have a good set of tests to know it still > works as intended. > > You'll want to do this even if OTP team go out of their way to make sure > everything stays compatible forever. > > I see no reason why we can't have both. What I am not okay with is: >> >> * Endlessly growing libraries of functions that are there for legacy >> reasons (wasting space and confusing new developers) >> * Fixing libraries over and over again because something got marked >> deprecated >> >> So how do we get both? >> > > You don't. > > In fact your second point is incorrect: there's absolutely no reason to > fix a library because something got marked deprecated. > > Zero. > > Developers should not "fix" libraries when something gets marked > deprecated. Developers should *take notice* that something has been > deprecated. > > There's nothing to fix to begin with! Check your OCD levels and stop > trying to get rid of all the deprecation warnings with obscure parse > transforms or version-specific defines... > > The time to fix comes when the feature gets *removed* or is rendered > *incompatible*. (Or soon before that actually happens, anyway.) > > As far as the string module is concerned, there's absolutely no need to > spend efforts to make your code run for both the old and the new string > module because the functions will not be removed for a very long time. And > until they are, chances are you will stop supporting OTP versions before > the new functions were introduced. You can do the changes THEN. > > I am baffled every time someone opens a ticket telling me that I have > deprecation warnings. I know! But they're harmless and the code works on > all supported versions, so why would I spend efforts to remove it? > > Your question can be reframed to be about an endlessly growing library + > having to fix libraries when things get removed, which is more of a choice, > you can't really have both. Either you have an endlessly growing library, > or you remove functions and fix libraries (manually or otherwise). That is what I meant. > > > -- > Lo?c Hoguin > https://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.des.courtis@REDACTED Fri Nov 24 19:39:33 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Fri, 24 Nov 2017 13:39:33 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> Message-ID: On Fri, Nov 24, 2017 at 9:40 AM, Lo?c Hoguin wrote: > On 11/24/2017 03:07 PM, Eric des Courtis wrote: > >> From my perspective, we want the following things: >> >> * The libraries to stay simple, small and clean (that means throwing >> away things) >> * Our old code to continue to work without modifications (that means >> having a mechanism for compiling old code targetting new versions of >> Erlang) >> > > That can work in many cases but not in the general case. This only works > for a specific set of changes where an equivalent still exists in the code > base. > That doesn't change the fact that most of the modules and functions we are talking about don't have this problem. Therefore it is probably an acceptable solution even if not perfect. > > For example removing ciphers or hashes in crypto requires human > intervention on every code bases that require them. > This is that is true and perhaps that is one of the few examples of where it doesn't make sence to try to preserve good backwards compatibility (assuming the app even knows about specific ciphers). > > Another issue comes from old code requiring syntax or VM features that > have been removed. > It doesn't have to be all or nothing. As far as syntax goes we can transpile the code. > > Then you have issues where it is technically still possible to do the old > functionality, but the performance or memory aspects are not equivalent. > Again in most cases that isn't really an issue. > > * To mix old and new Erlang code together >> > > If it's maintained, no problem. If not, even if it compiles and runs you > should probably make sure to have a good set of tests to know it still > works as intended. > Even if libraries are old they usually come with tests so it can work for many cases. > > You'll want to do this even if OTP team go out of their way to make sure > everything stays compatible forever. > We are already more or less in that boat already. > > I see no reason why we can't have both. What I am not okay with is: >> >> * Endlessly growing libraries of functions that are there for legacy >> reasons (wasting space and confusing new developers) >> * Fixing libraries over and over again because something got marked >> deprecated >> >> So how do we get both? > > You don't. > Yes but you can get something like 1.9 and reduce user pain substantially. > > In fact your second point is incorrect: there's absolutely no reason to > fix a library because something got marked deprecated. > There is because it usually gets removed (so do the work now or later what is the difference?). I don't always fix things until it's too late and things don't work. And too be honest this stuff happens too often in Erlang. > > Zero. > > Developers should not "fix" libraries when something gets marked > deprecated. Developers should *take notice* that something has been > deprecated. > If most deprecations never got removed I would agree however that isn't my experience. > > There's nothing to fix to begin with! Check your OCD levels and stop > trying to get rid of all the deprecation warnings with obscure parse > transforms or version-specific defines... That isn't really a big concern of mine. > > The time to fix comes when the feature gets *removed* or is rendered > *incompatible*. (Or soon before that actually happens, anyway.) > Yes and I keep finding myself forking all kinds of libraries to fix this crap. Often for no good reason at all. > > As far as the string module is concerned, there's absolutely no need to > spend efforts to make your code run for both the old and the new string > module because the functions will not be removed for a very long time. And > until they are, chances are you will stop supporting OTP versions before > the new functions were introduced. You can do the changes THEN. > Again it doesn't matter. You like doing changes last minute I don't. Now that doesn't mean I am going to change 10 libraries in my application to make sure they don't use tokens() or whatever. But when it gets removed I get to waste time chasing functions that have been removed. Instead of just marking it as OTP R16 compatible and moving on. The JVM solved this a long time ago with the "-source" and "-target" option. > > I am baffled every time someone opens a ticket telling me that I have > deprecation warnings. I know! But they're harmless and the code works on > all supported versions, so why would I spend efforts to remove it? > You wont get that kind of issue from me. > > Your question can be reframed to be about an endlessly growing library + > having to fix libraries when things get removed, which is more of a choice, > you can't really have both. Either you have an endlessly growing library, > or you remove functions and fix libraries (manually or otherwise). Yes that is what I meant to say (change deprecated to removed) . > > > -- > Lo?c Hoguin > https://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Nov 24 20:41:13 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Fri, 24 Nov 2017 20:41:13 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> Message-ID: <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> On 11/24/2017 07:39 PM, Eric des Courtis wrote: > > > On Fri, Nov 24, 2017 at 9:40 AM, Lo?c Hoguin > wrote: > > On 11/24/2017 03:07 PM, Eric des Courtis wrote: > > ?From my perspective, we want the following things: > > ? * The libraries to stay simple, small and clean (that means > throwing > ? ? away things) > ? * Our old code to continue to work without modifications > (that means > ? ? having a mechanism for compiling old code targetting new > versions of > ? ? Erlang) > > > That can work in many cases but not in the general case. This only > works for a specific set of changes where an equivalent still exists > in the code base. > > That doesn't change the fact that most of the modules and functions we > are talking about don't have this problem. Therefore it is probably an > acceptable solution even if not perfect. As far as OTP is concerned, this is not really the case. See https://github.com/erlang/otp/blob/master/lib/stdlib/src/otp_internal.erl Outside OTP, the code doesn't change too much and when it does it's generally for very good reasons (a simple mapping wouldn't help). > For example removing ciphers or hashes in crypto requires human > intervention on every code bases that require them. > > This is that is true and perhaps that is one of the few examples of > where it doesn't make sence to try to preserve good backwards > compatibility (assuming the app even knows about specific ciphers). > > > Another issue comes from old code requiring syntax or VM features > that have been removed. > > It doesn't have to be all or nothing. As far as syntax goes we can > transpile the code. Who's going to write the transpiler though? Better have OTP look toward the future than the past. > Then you have issues where it is technically still possible to do > the old functionality, but the performance or memory aspects are not > equivalent. > > Again in most cases that isn't really an issue. > > > ? * To mix old and new Erlang code together > > > If it's maintained, no problem. If not, even if it compiles and runs > you should probably make sure to have a good set of tests to know it > still works as intended. > > Even if libraries are old they usually come with tests so it can work > for many cases. > > > You'll want to do this even if OTP team go out of their way to make > sure everything stays compatible forever. > > We are already more or less in that boat already. That's pretty much my point. It doesn't really change things all that much. It's unclear whether the benefits outweight the efforts required. > I see no reason why we can't have both. What I am not okay with is: > > ? * Endlessly growing libraries of functions that are there for > legacy > ? ? reasons (wasting space and confusing new developers) > ? * Fixing libraries over and over again because something got > marked > ? ? deprecated > > So how do we get both? > > You don't. > > Yes but you can get something like 1.9 and reduce user pain substantially. > > > In fact your second point is incorrect: there's absolutely no reason > to fix a library because something got marked deprecated. > > There is because it usually gets removed (so do the work now or later > what is the difference?). I don't always fix things until it's too late > and things don't work. And too be honest this stuff happens too often in > Erlang. Well Fred's work around strings in rebar3 is a good example of where it does make a difference. Jumping through hoops to remove a warning when it could have easily been done a little before the functions were actually removed (but perhaps there was more value in this work than just the warning, I'm using it for the sake of an example). You could also see things like this done by people when rand was released. Well, random is still around. > Zero. > > Developers should not "fix" libraries when something gets marked > deprecated. Developers should *take notice* that something has been > deprecated. > > ?If most deprecations never got removed I would agree however that > isn't my experience. > > > There's nothing to fix to begin with! Check your OCD levels and stop > trying to get rid of all the deprecation warnings with obscure parse > transforms or version-specific defines... > > That isn't really a big concern of mine. > > > The time to fix comes when the feature gets *removed* or is rendered > *incompatible*. (Or soon before that actually happens, anyway.) > > Yes and I keep finding myself forking all kinds of libraries to fix this > crap. Often for no good reason at all. > > > As far as the string module is concerned, there's absolutely no need > to spend efforts to make your code run for both the old and the new > string module because the functions will not be removed for a very > long time. And until they are, chances are you will stop supporting > OTP versions before the new functions were introduced. You can do > the changes THEN. > > Again it doesn't matter. You like doing changes last minute I don't. Now > that doesn't mean I am going to change 10 libraries in my application to > make sure they don't use tokens() or whatever. But when it gets removed > I get to waste time chasing functions that have been removed. Instead of > just marking it as OTP R16 compatible and moving on. The JVM solved this > a long time ago with the "-source" and "-target" option. The JVM is a much better funded project with a lot more developers. It's hardly comparable. It's ultimately a question of priorities. OTP team could slow down adding features or improvements that break things in subtle ways (zlib NIF, recently) and focus on making sure code written 5 or 10 years ago can still run without change. Is it worth it? I doubt it. Especially since a lot of this software will slowly become incompatible with security practices, new protocols, new operating systems, new hardware, and so on and so forth. It's surely an interesting idea for the simpler cases, but it doesn't do much otherwise. > I am baffled every time someone opens a ticket telling me that I > have deprecation warnings. I know! But they're harmless and the code > works on all supported versions, so why would I spend efforts to > remove it? > > You wont get that kind of issue from me. > > > Your question can be reframed to be about an endlessly growing > library + having to fix libraries when things get removed, which is > more of a choice, you can't really have both. Either you have an > endlessly growing library, or you remove functions and fix libraries > (manually or otherwise). > > Yes that is what I meant to say (change deprecated to removed) . > > > > -- > Lo?c Hoguin > https://ninenines.eu > > -- Lo?c Hoguin https://ninenines.eu From ok@REDACTED Sat Nov 25 02:49:58 2017 From: ok@REDACTED (ok@REDACTED) Date: Sat, 25 Nov 2017 14:49:58 +1300 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: A couple of years ago I was working through a Java book. Not one of the examples got a clean compile. Not one. oddly enough, it was string-handling functions that had been deprecated, and oddly enough, in my environment the old functions still did everything I needed. It's not just Erlang and Java. I had C code using some of the classic string functions in ways I had carefully ensured were correct. Porting to the next version of the Unix flavour I was using, the linker screamed at me about unsafe functions. Since I wanted other people to use the program and didn't want them looking down on me, I spent a merry couple of hours changing the code to use memcpy instead of strcpy and so on. Again, I have a program which works fine in Solaris 10, Open Solaris, and Solaris 11 Express. It uses a mix of old functions (hey, if the code still works, why change it, am I right?) and new functions (if you can call POSIX 2008 "new".) But OpenIndiana? I am still baffled as to what combination of feature test macros I can set to make the program compile, and am coming to the conclusion that there isn't one. Did I mention the trouble I've been having with Ubuntu 17.10? I'll spare you that, but let's just say that putting standard headers in nonstandard places really really does not help. I don't really have a solution. It seems as though the only thing you can do to ensure that old code continues to work is to keep a VM imagine with a complete copy of the environment it used to work in. Good luck plugging new libraries into that, though. From nistrigunya@REDACTED Sat Nov 25 04:39:29 2017 From: nistrigunya@REDACTED (Avinash Dhumane) Date: Sat, 25 Nov 2017 09:09:29 +0530 Subject: [erlang-questions] Wrap my mind around NoSQL (Mnesia) In-Reply-To: <20171124145038.GA23275@auron> References: <20171124145038.GA23275@auron> Message-ID: Separate (and postpone) the data representation (i.e. tables) concerns from the data access (CRUD). The access needs of the application will determine the tables (whether set or bag or ordered) and objects & attributes (keys and values). That is, form follows the function. On Fri, Nov 24, 2017 at 8:20 PM, Silas wrote: > Hi! > > (This is mainly a question about NoSQL conceptions. If it is the wrong > place to ask, sorry for that and, please, let me know). > > After years using RDBMS (with structured and OO paradigms at the > programming language side) I'm diving into functional programming. When > choosing a programming language to practice, I immediately fell in love > with Erlang for this simplicity and beauty. > > After reading some documentation, some chapters of a book and trying > simple projects I decided to try a more complex project with Mnesia. It is > a social bookmarks website (something similar to what del.icio.us is) > where user can bookmark a URL and associate one or more tags with it. For > instance, some could bookmark "erlang.org" to tags > "functionalprogramming", "programminglanguage" and others. Users also can > query other users bookmarks, see what links are associated with a tag and > what tags are associated with a given bookmark. > > In my mind, an RDBMS is a perfect solution for this problem. I'm trying > to fit it into the key -> value model, though. My first question is: is > this possible or I'm just doing the wrong thing by forcing the wrong > solution? > > For what I understood, I need to get rid of normalization rules. So I > devised something more or less like that: > > User table: {username, {urls, [tags]}} % set > UserTag table: {user, [tags]} % set > TagUrls table: {tag, url} % bag > TagUsers table: {tag, user} % bag > Urlusers table: {url, user} %bag > UrlTags table: {url, tag} %bag > > It is still a bit confusing where I'd use a set, a bag or an ordered_set > but I think you got the whole picture. > > Any help? > > Thanks! > > -- > Silas > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sat Nov 25 15:10:00 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 25 Nov 2017 14:10:00 +0000 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: I think the only way to approach a problem like this is by asking: "How much are you willing to spend on your backwards compatibility?" Maintaining a long-term-release of a piece of software is no easy feat. It requires effort, and that requires money. If you need stability from an older release and you need control of that release, you have to give the right people some money. In Solaris, Sun had the advantage that they could use the Zone construction in the OS. You could brand a zone, such that inside the zone, the Solaris (10) system operates as if it were a Solaris 8 or 9 system. This is achieved by manipulating the syscall table of the kernel and handling some syscalls differently. Maintenance of such a compat layer is time-consuming, however. In FreeBSD, you have libcompat for much the same thing: It provides you with a way to run legacy binaries on a new system for a while, so you don't have to upgrade all of your software in one go. In Android development, the system runs by "API levels" (corresponding somewhat to Erlang/OTP major releases). A given library functionality is *introduced* at some API level, *deprecated* at a higher level and eventually *removed* in an even higher API level. This means that any function is alive over a "window" from introduction to removal. In practice, one has to cope with changes of a system over time. What is a problem is the rate-of-change, not the change itself. Historically, Erlang is a fairly slow-changing system I think. My older Haskell code rarely compiles, and the recent changes in OCaml also requires far more interaction to make things work again on some code bases. The changes are likely to require library maintainers to handle several releases and this is where the hard part of the work is. A single project can target a single Erlang release. A library, which may have to span several Erlang releases * Cannot use a new function when it is introduced. That would break backwards compatibility. * Cannot use functions which have been removed. That would break forwards compatibility. How large a deprecation window can be depends largely on factors such as "how common is the function, and how easy is it to change the code?", "Do we have a sponsor who wishes the older versions kept around, paying us to do so?", and so on. You may quickly find it is cheaper for you to upgrade the Erlang/OTP release rather than maintaining and older version of it. On Sat, Nov 25, 2017 at 2:50 AM wrote: > A couple of years ago I was working through a Java book. > Not one of the examples got a clean compile. Not one. > oddly enough, it was string-handling functions that had > been deprecated, and oddly enough, in my environment the > old functions still did everything I needed. > > It's not just Erlang and Java. I had C code using some of > the classic string functions in ways I had carefully ensured > were correct. Porting to the next version of the Unix > flavour I was using, the linker screamed at me about unsafe > functions. Since I wanted other people to use the program > and didn't want them looking down on me, I spent a merry > couple of hours changing the code to use memcpy instead of > strcpy and so on. > > Again, I have a program which works fine in Solaris 10, Open > Solaris, and Solaris 11 Express. It uses a mix of old > functions (hey, if the code still works, why change it, am I > right?) and new functions (if you can call POSIX 2008 "new".) > But OpenIndiana? I am still baffled as to what combination of > feature test macros I can set to make the program compile, and > am coming to the conclusion that there isn't one. > > Did I mention the trouble I've been having with Ubuntu 17.10? > I'll spare you that, but let's just say that putting standard > headers in nonstandard places really really does not help. > > I don't really have a solution. It seems as though the only > thing you can do to ensure that old code continues to work is > to keep a VM imagine with a complete copy of the environment > it used to work in. > > Good luck plugging new libraries into that, though. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sat Nov 25 15:17:27 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 25 Nov 2017 14:17:27 +0000 Subject: [erlang-questions] EEP 47: Syntax in try/catch to retrieve the stacktrace directly In-Reply-To: References: Message-ID: I think this is a sensible change all in all, and one which should be done. My major gripe with it is the fact that you cannot pattern match on the stack trace. I see why this is done, but it feels like being a special arbitrary rule which goes against other matching patterns in the language. I'm not sure how it would help catch Cl:Err:Stk -> case {Cl, Err, Stk} of ... end What I really like about the change is that it removes get_stacktrace/0 which is a "bad function", and one of those things I tend to call a "language mistake" in Erlang[0]. [0] For the interested, I think one of the most obvious mistakes are that a floating point numbers in Erlang cannot express NaNs and Infinities. On Fri, Nov 24, 2017 at 10:05 AM Bj?rn Gustavsson wrote: > There is a new EEP that proposes a new syntax in try/catch to retrieve > the stacktrace directly without using erlang:get_stacktrace/0: > > https://github.com/erlang/eep/blob/master/eeps/eep-0047.md > > /Bj?rn > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Sat Nov 25 16:00:35 2017 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Sat, 25 Nov 2017 10:00:35 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: A perhaps all-to-naive suggestion: 1. Suppose we move all new Unicode functions to a new module called ustring, unicode, or some such. 2. Keep the string module as is, but explicitly typespec every function. I don't have time to check if Erlang has a cstring type, but if not, create it. All the best, Lloyd Sent from my iPad > On Nov 25, 2017, at 9:10 AM, Jesper Louis Andersen wrote: > > I think the only way to approach a problem like this is by asking: > > "How much are you willing to spend on your backwards compatibility?" > > Maintaining a long-term-release of a piece of software is no easy feat. It requires effort, and that requires money. If you need stability from an older release and you need control of that release, you have to give the right people some money. > > In Solaris, Sun had the advantage that they could use the Zone construction in the OS. You could brand a zone, such that inside the zone, the Solaris (10) system operates as if it were a Solaris 8 or 9 system. This is achieved by manipulating the syscall table of the kernel and handling some syscalls differently. Maintenance of such a compat layer is time-consuming, however. In FreeBSD, you have libcompat for much the same thing: It provides you with a way to run legacy binaries on a new system for a while, so you don't have to upgrade all of your software in one go. > > In Android development, the system runs by "API levels" (corresponding somewhat to Erlang/OTP major releases). A given library functionality is *introduced* at some API level, *deprecated* at a higher level and eventually *removed* in an even higher API level. This means that any function is alive over a "window" from introduction to removal. > > In practice, one has to cope with changes of a system over time. What is a problem is the rate-of-change, not the change itself. Historically, Erlang is a fairly slow-changing system I think. My older Haskell code rarely compiles, and the recent changes in OCaml also requires far more interaction to make things work again on some code bases. > > The changes are likely to require library maintainers to handle several releases and this is where the hard part of the work is. A single project can target a single Erlang release. A library, which may have to span several Erlang releases > > * Cannot use a new function when it is introduced. That would break backwards compatibility. > * Cannot use functions which have been removed. That would break forwards compatibility. > > How large a deprecation window can be depends largely on factors such as "how common is the function, and how easy is it to change the code?", "Do we have a sponsor who wishes the older versions kept around, paying us to do so?", and so on. You may quickly find it is cheaper for you to upgrade the Erlang/OTP release rather than maintaining and older version of it. > > > >> On Sat, Nov 25, 2017 at 2:50 AM wrote: >> A couple of years ago I was working through a Java book. >> Not one of the examples got a clean compile. Not one. >> oddly enough, it was string-handling functions that had >> been deprecated, and oddly enough, in my environment the >> old functions still did everything I needed. >> >> It's not just Erlang and Java. I had C code using some of >> the classic string functions in ways I had carefully ensured >> were correct. Porting to the next version of the Unix >> flavour I was using, the linker screamed at me about unsafe >> functions. Since I wanted other people to use the program >> and didn't want them looking down on me, I spent a merry >> couple of hours changing the code to use memcpy instead of >> strcpy and so on. >> >> Again, I have a program which works fine in Solaris 10, Open >> Solaris, and Solaris 11 Express. It uses a mix of old >> functions (hey, if the code still works, why change it, am I >> right?) and new functions (if you can call POSIX 2008 "new".) >> But OpenIndiana? I am still baffled as to what combination of >> feature test macros I can set to make the program compile, and >> am coming to the conclusion that there isn't one. >> >> Did I mention the trouble I've been having with Ubuntu 17.10? >> I'll spare you that, but let's just say that putting standard >> headers in nonstandard places really really does not help. >> >> I don't really have a solution. It seems as though the only >> thing you can do to ensure that old code continues to work is >> to keep a VM imagine with a complete copy of the environment >> it used to work in. >> >> Good luck plugging new libraries into that, though. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Sat Nov 25 16:11:23 2017 From: kostis@REDACTED (Kostis Sagonas) Date: Sat, 25 Nov 2017 16:11:23 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: On 11/25/2017 04:00 PM, Lloyd R. Prentice wrote: > > ?I don't have time to check if Erlang has a cstring type, but if not, > create it. What _exactly_ is a cstring and what does it have to do with Erlang? Kostis From attila.r.nohl@REDACTED Sun Nov 26 14:40:19 2017 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Sun, 26 Nov 2017 14:40:19 +0100 Subject: [erlang-questions] EEP 47: Syntax in try/catch to retrieve the stacktrace directly In-Reply-To: References: Message-ID: Once I worked with a library that implemented the "let it crash" philosophy to the maximum - for valid input I got the result, for invalid input I got function_clause, badarg, badmatch, etc. crashes. I tried to be clever and match on the output of erlang:get_stacktrace() to decide if the error came from this particular library or from a different code (in order to provide more meaningful error message to the user), but eventually it didn't work out. In that case matching on the stack trace might have been useful, but as it didn't work out in the end, I don't think it's such a big deal that I can't match on the stacktrace. 2017-11-25 15:17 GMT+01:00 Jesper Louis Andersen : > I think this is a sensible change all in all, and one which should be done. > > My major gripe with it is the fact that you cannot pattern match on the > stack trace. I see why this is done, but it feels like being a special > arbitrary rule which goes against other matching patterns in the language. > I'm not sure how it would help > > catch > Cl:Err:Stk -> > case {Cl, Err, Stk} of > ... > end > > What I really like about the change is that it removes get_stacktrace/0 > which is a "bad function", and one of those things I tend to call a > "language mistake" in Erlang[0]. > > [0] For the interested, I think one of the most obvious mistakes are that a > floating point numbers in Erlang cannot express NaNs and Infinities. > > On Fri, Nov 24, 2017 at 10:05 AM Bj?rn Gustavsson wrote: >> >> There is a new EEP that proposes a new syntax in try/catch to retrieve >> the stacktrace directly without using erlang:get_stacktrace/0: >> >> https://github.com/erlang/eep/blob/master/eeps/eep-0047.md >> >> /Bj?rn >> -- >> Bj?rn Gustavsson, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From lloyd@REDACTED Sun Nov 26 19:33:57 2017 From: lloyd@REDACTED (lloyd@REDACTED) Date: Sun, 26 Nov 2017 13:33:57 -0500 (EST) Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: <1511721237.783226924@apps.rackspace.com> Hi Kostis, This thread has explored the policy of deprecating and dropping functions from the string library. Arguments have been made for and against. The term "cstring" was mentioned in a post long gone by. Perhaps it is not a technical term in any language, and I have inadvertently introduced confusion. But I take cstring to mean a list of ASCII characters. The suggestion in my previous post was aimed at finding a way to move forward with strong support for Unicode while maintaining a clear distinction between the new Unicode string functions and the currently deprecated string functions. We all agree, I believe, that we'd rather not have a language cluttered up with functions that were once useful but have fallen behind advancing technology. And some posts in this thread contend that the deprecated string functions fall into this category. Thus, they favor a policy that deprecates them and marks them for eventual elimination from the language. The devil in the details here is the word "eventual." But I argue that the currently deprecated functions will be useful as long as the English language is widely used around the world. So why not find a way to clearly distinguish between the deprecated string functions and the Unicode functions without breaking legacy code? Since at this time, I'm guessing, that there is more code in the wild that incorporate the deprecated functions than the Unicode functions, then putting the Unicode functions in a separate module may impose less of a burden on the community. Plus, if and when , if ever, there is strong consensus that the deprecated string functions are of little use to anybody in the Erlang user community, the module as a whole can be dropped from Erlang. Best wishes, Lloyd -----Original Message----- From: "Kostis Sagonas" Sent: Saturday, November 25, 2017 10:11am To: erlang-questions@REDACTED Subject: Re: [erlang-questions] Strings - deprecated functions On 11/25/2017 04:00 PM, Lloyd R. Prentice wrote: > > I don't have time to check if Erlang has a cstring type, but if not, > create it. What _exactly_ is a cstring and what does it have to do with Erlang? Kostis _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott_ribe@REDACTED Sun Nov 26 20:20:46 2017 From: scott_ribe@REDACTED (scott ribe) Date: Sun, 26 Nov 2017 12:20:46 -0700 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1511721237.783226924@apps.rackspace.com> References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> <1511721237.783226924@apps.rackspace.com> Message-ID: On Nov 26, 2017, at 11:33 AM, lloyd@REDACTED wrote: > > But I argue that the currently deprecated functions will be useful as long as the English language is widely used around the world. But they are not MORE useful than the new functions for English, since UTF-8 is a strict superset of ASCII. -- Scott Ribe https://www.linkedin.com/in/scottribe/ (303) 722-0567 From rvirding@REDACTED Mon Nov 27 03:03:06 2017 From: rvirding@REDACTED (Robert Virding) Date: Mon, 27 Nov 2017 03:03:06 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: I think it is perfectly possible to keep the old functions in the same 'string' module in a clean way if you just reorganise the documentation. Now all the functions, both the new and the obsolete, are kept in one alphabetical list which makes it difficult to see which are which and get a clear overview of the functions. Why not just separate the functions into 2 groups, the new and the obsolete, each in its own alphabetical list? This would make it easy to keep the old functions while making it easy for people to choose the newer ones, and migrate code if desired. There would then be no need to get rid of the old functions and break backward compatibility. Robert On 25 November 2017 at 15:10, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > I think the only way to approach a problem like this is by asking: > > "How much are you willing to spend on your backwards compatibility?" > > Maintaining a long-term-release of a piece of software is no easy feat. It > requires effort, and that requires money. If you need stability from an > older release and you need control of that release, you have to give the > right people some money. > > In Solaris, Sun had the advantage that they could use the Zone > construction in the OS. You could brand a zone, such that inside the zone, > the Solaris (10) system operates as if it were a Solaris 8 or 9 system. > This is achieved by manipulating the syscall table of the kernel and > handling some syscalls differently. Maintenance of such a compat layer is > time-consuming, however. In FreeBSD, you have libcompat for much the same > thing: It provides you with a way to run legacy binaries on a new system > for a while, so you don't have to upgrade all of your software in one go. > > In Android development, the system runs by "API levels" (corresponding > somewhat to Erlang/OTP major releases). A given library functionality is > *introduced* at some API level, *deprecated* at a higher level and > eventually *removed* in an even higher API level. This means that any > function is alive over a "window" from introduction to removal. > > In practice, one has to cope with changes of a system over time. What is a > problem is the rate-of-change, not the change itself. Historically, Erlang > is a fairly slow-changing system I think. My older Haskell code rarely > compiles, and the recent changes in OCaml also requires far more > interaction to make things work again on some code bases. > > The changes are likely to require library maintainers to handle several > releases and this is where the hard part of the work is. A single project > can target a single Erlang release. A library, which may have to span > several Erlang releases > > * Cannot use a new function when it is introduced. That would break > backwards compatibility. > * Cannot use functions which have been removed. That would break forwards > compatibility. > > How large a deprecation window can be depends largely on factors such as > "how common is the function, and how easy is it to change the code?", "Do > we have a sponsor who wishes the older versions kept around, paying us to > do so?", and so on. You may quickly find it is cheaper for you to upgrade > the Erlang/OTP release rather than maintaining and older version of it. > > > > On Sat, Nov 25, 2017 at 2:50 AM wrote: > >> A couple of years ago I was working through a Java book. >> Not one of the examples got a clean compile. Not one. >> oddly enough, it was string-handling functions that had >> been deprecated, and oddly enough, in my environment the >> old functions still did everything I needed. >> >> It's not just Erlang and Java. I had C code using some of >> the classic string functions in ways I had carefully ensured >> were correct. Porting to the next version of the Unix >> flavour I was using, the linker screamed at me about unsafe >> functions. Since I wanted other people to use the program >> and didn't want them looking down on me, I spent a merry >> couple of hours changing the code to use memcpy instead of >> strcpy and so on. >> >> Again, I have a program which works fine in Solaris 10, Open >> Solaris, and Solaris 11 Express. It uses a mix of old >> functions (hey, if the code still works, why change it, am I >> right?) and new functions (if you can call POSIX 2008 "new".) >> But OpenIndiana? I am still baffled as to what combination of >> feature test macros I can set to make the program compile, and >> am coming to the conclusion that there isn't one. >> >> Did I mention the trouble I've been having with Ubuntu 17.10? >> I'll spare you that, but let's just say that putting standard >> headers in nonstandard places really really does not help. >> >> I don't really have a solution. It seems as though the only >> thing you can do to ensure that old code continues to work is >> to keep a VM imagine with a complete copy of the environment >> it used to work in. >> >> Good luck plugging new libraries into that, though. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Nov 27 04:02:58 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Sun, 26 Nov 2017 22:02:58 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: <20171127030257.GG2024@ferdmbp.local> On 11/27, Robert Virding wrote: >I think it is perfectly possible to keep the old functions in the same >'string' module in a clean way if you just reorganise the documentation. >Now all the functions, both the new and the obsolete, are kept in one >alphabetical list which makes it difficult to see which are which and get a >clear overview of the functions. Why not just separate the functions into 2 >groups, the new and the obsolete, each in its own alphabetical list? This >would make it easy to keep the old functions while making it easy for >people to choose the newer ones, and migrate code if desired. There would >then be no need to get rid of the old functions and break backward >compatibility. > The index on the left of the docs is alphabetical (that's automated), but the functions on the page itself are already in a new/old set of functions, each sorted alphabetically. From ok@REDACTED Mon Nov 27 08:06:49 2017 From: ok@REDACTED (ok@REDACTED) Date: Mon, 27 Nov 2017 20:06:49 +1300 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: <162e7a61217974fcaea64106a5f5513b.squirrel@chasm.otago.ac.nz> An alphabetic list has two uses: 1. find a function with (approximately) known semantics but unknown name 2. find the documentation for a known name. Splitting the documentation into two lists hurts 2 without actually helping 1. A more informative approach would be to have a line added: 12.3 deprecated: 20.3 to be removed: 24.0 perhaps with dates, and it should be possible to generate at least the 'added:' and 'deprecated:' parts automatically. "added" = JavaDoc "since". Compare also the macOS X __OSX_AVAILABLE_STARTING(, ) __OSX_AVAILABLE_BUT_DEPRECATED(, , , ) From cean.ebengt@REDACTED Mon Nov 27 08:15:28 2017 From: cean.ebengt@REDACTED (bengt e) Date: Mon, 27 Nov 2017 08:15:28 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: Greetings, It would be nice if Erlang modules where prefixed with their app. The new functions would then be in stdlib_strings, instead of string. I know that this would only help once, ie when functions in stdlib_strings are deprecated we would have the current problem again. But that would be in 20 years time, right? Perhaps a better idea has come up by then. bengt On Mon, Nov 27, 2017 at 3:03 AM, Robert Virding wrote: > I think it is perfectly possible to keep the old functions in the same > 'string' module in a clean way if you just reorganise the documentation. > Now all the functions, both the new and the obsolete, are kept in one > alphabetical list which makes it difficult to see which are which and get a > clear overview of the functions. Why not just separate the functions into 2 > groups, the new and the obsolete, each in its own alphabetical list? This > would make it easy to keep the old functions while making it easy for > people to choose the newer ones, and migrate code if desired. There would > then be no need to get rid of the old functions and break backward > compatibility. > > Robert > > > On 25 November 2017 at 15:10, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED> wrote: > >> I think the only way to approach a problem like this is by asking: >> >> "How much are you willing to spend on your backwards compatibility?" >> >> Maintaining a long-term-release of a piece of software is no easy feat. >> It requires effort, and that requires money. If you need stability from an >> older release and you need control of that release, you have to give the >> right people some money. >> >> In Solaris, Sun had the advantage that they could use the Zone >> construction in the OS. You could brand a zone, such that inside the zone, >> the Solaris (10) system operates as if it were a Solaris 8 or 9 system. >> This is achieved by manipulating the syscall table of the kernel and >> handling some syscalls differently. Maintenance of such a compat layer is >> time-consuming, however. In FreeBSD, you have libcompat for much the same >> thing: It provides you with a way to run legacy binaries on a new system >> for a while, so you don't have to upgrade all of your software in one go. >> >> In Android development, the system runs by "API levels" (corresponding >> somewhat to Erlang/OTP major releases). A given library functionality is >> *introduced* at some API level, *deprecated* at a higher level and >> eventually *removed* in an even higher API level. This means that any >> function is alive over a "window" from introduction to removal. >> >> In practice, one has to cope with changes of a system over time. What is >> a problem is the rate-of-change, not the change itself. Historically, >> Erlang is a fairly slow-changing system I think. My older Haskell code >> rarely compiles, and the recent changes in OCaml also requires far more >> interaction to make things work again on some code bases. >> >> The changes are likely to require library maintainers to handle several >> releases and this is where the hard part of the work is. A single project >> can target a single Erlang release. A library, which may have to span >> several Erlang releases >> >> * Cannot use a new function when it is introduced. That would break >> backwards compatibility. >> * Cannot use functions which have been removed. That would break forwards >> compatibility. >> >> How large a deprecation window can be depends largely on factors such as >> "how common is the function, and how easy is it to change the code?", "Do >> we have a sponsor who wishes the older versions kept around, paying us to >> do so?", and so on. You may quickly find it is cheaper for you to upgrade >> the Erlang/OTP release rather than maintaining and older version of it. >> >> >> >> On Sat, Nov 25, 2017 at 2:50 AM wrote: >> >>> A couple of years ago I was working through a Java book. >>> Not one of the examples got a clean compile. Not one. >>> oddly enough, it was string-handling functions that had >>> been deprecated, and oddly enough, in my environment the >>> old functions still did everything I needed. >>> >>> It's not just Erlang and Java. I had C code using some of >>> the classic string functions in ways I had carefully ensured >>> were correct. Porting to the next version of the Unix >>> flavour I was using, the linker screamed at me about unsafe >>> functions. Since I wanted other people to use the program >>> and didn't want them looking down on me, I spent a merry >>> couple of hours changing the code to use memcpy instead of >>> strcpy and so on. >>> >>> Again, I have a program which works fine in Solaris 10, Open >>> Solaris, and Solaris 11 Express. It uses a mix of old >>> functions (hey, if the code still works, why change it, am I >>> right?) and new functions (if you can call POSIX 2008 "new".) >>> But OpenIndiana? I am still baffled as to what combination of >>> feature test macros I can set to make the program compile, and >>> am coming to the conclusion that there isn't one. >>> >>> Did I mention the trouble I've been having with Ubuntu 17.10? >>> I'll spare you that, but let's just say that putting standard >>> headers in nonstandard places really really does not help. >>> >>> I don't really have a solution. It seems as though the only >>> thing you can do to ensure that old code continues to work is >>> to keep a VM imagine with a complete copy of the environment >>> it used to work in. >>> >>> Good luck plugging new libraries into that, though. >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Mon Nov 27 09:41:20 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 27 Nov 2017 09:41:20 +0100 Subject: [erlang-questions] EEP 47: Syntax in try/catch to retrieve the stacktrace directly In-Reply-To: References: Message-ID: I agree that this is a good change. I can't even recall exactly why we went with the get_stacktrace() BIF - particularly since the value is already present as a variable under the hood. But I guess it was precisely to avoid such discussions about pattern matching on stack traces or having arbitrary restrictions such as "may only be a variable" on a part of a pattern. But I think such a restriction makes sense. The main problem with code that matches on the format of the stack trace is that it creates a hard dependency on the current format of the stack trace - which has changed more than once and might change again. The general rule is Don't Do That, and if you still feel that you have to, at least limit it to a single support function, not spread all over your code. /Richard 2017-11-26 14:40 GMT+01:00 Attila Rajmund Nohl : > Once I worked with a library that implemented the "let it crash" > philosophy to the maximum - for valid input I got the result, for > invalid input I got function_clause, badarg, badmatch, etc. crashes. I > tried to be clever and match on the output of erlang:get_stacktrace() > to decide if the error came from this particular library or from a > different code (in order to provide more meaningful error message to > the user), but eventually it didn't work out. In that case matching on > the stack trace might have been useful, but as it didn't work out in > the end, I don't think it's such a big deal that I can't match on the > stacktrace. > > 2017-11-25 15:17 GMT+01:00 Jesper Louis Andersen > : > > I think this is a sensible change all in all, and one which should be > done. > > > > My major gripe with it is the fact that you cannot pattern match on the > > stack trace. I see why this is done, but it feels like being a special > > arbitrary rule which goes against other matching patterns in the > language. > > I'm not sure how it would help > > > > catch > > Cl:Err:Stk -> > > case {Cl, Err, Stk} of > > ... > > end > > > > What I really like about the change is that it removes get_stacktrace/0 > > which is a "bad function", and one of those things I tend to call a > > "language mistake" in Erlang[0]. > > > > [0] For the interested, I think one of the most obvious mistakes are > that a > > floating point numbers in Erlang cannot express NaNs and Infinities. > > > > On Fri, Nov 24, 2017 at 10:05 AM Bj?rn Gustavsson > wrote: > >> > >> There is a new EEP that proposes a new syntax in try/catch to retrieve > >> the stacktrace directly without using erlang:get_stacktrace/0: > >> > >> https://github.com/erlang/eep/blob/master/eeps/eep-0047.md > >> > >> /Bj?rn > >> -- > >> Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fxn@REDACTED Mon Nov 27 10:10:39 2017 From: fxn@REDACTED (Xavier Noria) Date: Mon, 27 Nov 2017 10:10:39 +0100 Subject: [erlang-questions] questions about system events Message-ID: The section "System Messages" of Cesarini & Vinosky explains there are three types of system messages: {in, Msg} {out, Msg, To, State} term() and mentions that the docs up to Erlang 18 also specify {in, Msg, From} {out, Msg, To} but they are (were) not used by any standard behaviors. I have compared this to the current docs ( http://erlang.org/doc/man/sys.html#type-system_event) and see a few things: * Current docs use the term "system event" rather than "system message". Is that a change in terminology that happened at some point? * I guess the API has changed since Erlang 18 and {out, Msg, To, State} is no longer documented, there's {out, Msg, To} today. Are you supposed to issue additional calls in order to trace state? * {in, Msg, From} is still documented, is now used by some standard behavior? -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik.x.nord@REDACTED Mon Nov 27 11:26:48 2017 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Mon, 27 Nov 2017 10:26:48 +0000 Subject: [erlang-questions] Contributor License Agreement for pullrequests Message-ID: <1511778407.3773.12.camel@ericsson.com> Hello Erlang Questions! As some of you already noticed, we have now added a CLA application for github pullrequests. This is done to ensure that we (Erlang/OTP at Ericsson) always have a up to date list of contributors, and that you contribute code that you are yours to commit, and free for us to use and maintain under the?Apache License 2.0. If and When you submit a pullrequest for the first time (after this change) you will be prompted via e-mail and a comment on your pullrequest to visit the?https://cla-assistant.io/?and agree to this. We will then store your github username, first and last name and email in much the same way as before. BR Henrik Nord Erlang/OTP From bjorn@REDACTED Mon Nov 27 15:22:49 2017 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 27 Nov 2017 15:22:49 +0100 Subject: [erlang-questions] EEP 47: Syntax in try/catch to retrieve the stacktrace directly In-Reply-To: References: Message-ID: On Sat, Nov 25, 2017 at 3:17 PM, Jesper Louis Andersen wrote: [...] > > My major gripe with it is the fact that you cannot pattern match on the > stack trace. Yes, I don't like that inconsistency myself, but I think that the alternatives are worse. If we were to implement pattern matching on the stacktrace, it would be much slower than the matching the pattern for the exception. I find that kind of performance trap to be worse than not allowing matching in the first place. I have updated the pull request so that there is now a compilation error if the variable for the stacktrace is already bound. The compiler already gives an error if an attempt is made to use the stacktrace variable in the guard. Since the compiler is that strict, it would be a compatible change to implement matching in the future, if it turns out that is desirable or reduces user confusion. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From bjorn@REDACTED Mon Nov 27 15:26:37 2017 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 27 Nov 2017 15:26:37 +0100 Subject: [erlang-questions] EEP 47: Syntax in try/catch to retrieve the stacktrace directly In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 9:41 AM, Richard Carlsson wrote: > I agree that this is a good change. I can't even recall exactly why we went > with the get_stacktrace() BIF - particularly since the value is already > present as a variable under the hood. But I guess it was precisely to avoid > such discussions about pattern matching on stack traces or having arbitrary > restrictions such as "may only be a variable" on a part of a pattern. But I > think such a restriction makes sense. I have a vague memory that it was also problems with the parser. I had to to refactor the yecc rules a bit to get it to work. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From jesper.louis.andersen@REDACTED Mon Nov 27 15:31:35 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 27 Nov 2017 14:31:35 +0000 Subject: [erlang-questions] EEP 47: Syntax in try/catch to retrieve the stacktrace directly In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 3:22 PM Bj?rn Gustavsson wrote: > On Sat, Nov 25, 2017 at 3:17 PM, Jesper Louis Andersen > wrote: > [...] > > > > My major gripe with it is the fact that you cannot pattern match on the > > stack trace. > > Yes, I don't like that inconsistency myself, but I > think that the alternatives are worse. > > Yes, I think so too. In a typed language, you would probably declare an abstract type for the stack and not provide any kind of matching pattern for it. This would force people to handle the stack by printing, and there would be no matching on it at all. Mimicking this behavior in Erlang is probably the sane behavior in this case. I also like Richard's point: matching on the stack will eventually get you into trouble. -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila.r.nohl@REDACTED Mon Nov 27 16:08:47 2017 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Mon, 27 Nov 2017 16:08:47 +0100 Subject: [erlang-questions] EEP 47: Syntax in try/catch to retrieve the stacktrace directly In-Reply-To: References: Message-ID: 2017-11-27 15:31 GMT+01:00 Jesper Louis Andersen : > On Mon, Nov 27, 2017 at 3:22 PM Bj?rn Gustavsson wrote: >> >> On Sat, Nov 25, 2017 at 3:17 PM, Jesper Louis Andersen >> wrote: >> [...] >> > >> > My major gripe with it is the fact that you cannot pattern match on the >> > stack trace. >> >> Yes, I don't like that inconsistency myself, but I >> think that the alternatives are worse. >> > > Yes, I think so too. In a typed language, you would probably declare an > abstract type for the stack and not provide any kind of matching pattern for > it. This would force people to handle the stack by printing, and there would > be no matching on it at all. > > Mimicking this behavior in Erlang is probably the sane behavior in this > case. If the stack trace is strictly for human consumption, why not return a string (list or binary)? From n.oxyde@REDACTED Mon Nov 27 18:01:31 2017 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 27 Nov 2017 18:01:31 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> Message-ID: <7A89A617-498E-4423-9590-B491B31BBB34@gmail.com> I don't particularly care about the old functions myself (if you want to run old code, just run a damn old VM, problem solved), but a way to reorganise the documentation to accommodate for that would also bring closure to https://github.com/erlang/otp/pull/363, which would be nice. > Le 27 nov. 2017 ? 03:03, Robert Virding a ?crit : > > I think it is perfectly possible to keep the old functions in the same 'string' module in a clean way if you just reorganise the documentation. Now all the functions, both the new and the obsolete, are kept in one alphabetical list which makes it difficult to see which are which and get a clear overview of the functions. Why not just separate the functions into 2 groups, the new and the obsolete, each in its own alphabetical list? This would make it easy to keep the old functions while making it easy for people to choose the newer ones, and migrate code if desired. There would then be no need to get rid of the old functions and break backward compatibility. From n.oxyde@REDACTED Mon Nov 27 18:06:56 2017 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 27 Nov 2017 18:06:56 +0100 Subject: [erlang-questions] Contributor License Agreement for pullrequests In-Reply-To: <1511778407.3773.12.camel@ericsson.com> References: <1511778407.3773.12.camel@ericsson.com> Message-ID: <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> What's the rationale for this? > Le 27 nov. 2017 ? 11:26, Henrik Nord X a ?crit : > > Hello Erlang Questions! > > > As some of you already noticed, we have now added a CLA application for > github pullrequests. > > This is done to ensure that we (Erlang/OTP at Ericsson) always have a > up to date list of contributors, and that you contribute code that you > are yours to commit, and free for us to use and maintain under > the Apache License 2.0. > > If and When you submit a pullrequest for the first time (after this > change) you will be prompted via e-mail and a comment on your > pullrequest to visit the https://cla-assistant.io/ and agree to this. > > We will then store your github username, first and last name and email > in much the same way as before. > > BR > Henrik Nord > Erlang/OTP > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Mon Nov 27 22:22:12 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 27 Nov 2017 22:22:12 +0100 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <7A89A617-498E-4423-9590-B491B31BBB34@gmail.com> References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> <7A89A617-498E-4423-9590-B491B31BBB34@gmail.com> Message-ID: <1e3ae868-5b0f-e99d-c672-784164d1cb45@ninenines.eu> The reason this PR was closed is a little ridiculous. stdlib has erl_internal and other modules before most modules that are useful to most people and the important modules are effectively hidden in this confusing mess of useful and not-so-useful-at-all modules. Meanwhile the compiler application has exactly one module right now; you're not going to confuse anyone by adding a couple more modules. Ridiculous. On 11/27/2017 06:01 PM, Anthony Ramine wrote: > I don't particularly care about the old functions myself (if you want to run old code, just run a damn old VM, problem solved), but a way to reorganise the documentation to accommodate for that would also bring closure to https://github.com/erlang/otp/pull/363, which would be nice. > >> Le 27 nov. 2017 ? 03:03, Robert Virding a ?crit : >> >> I think it is perfectly possible to keep the old functions in the same 'string' module in a clean way if you just reorganise the documentation. Now all the functions, both the new and the obsolete, are kept in one alphabetical list which makes it difficult to see which are which and get a clear overview of the functions. Why not just separate the functions into 2 groups, the new and the obsolete, each in its own alphabetical list? This would make it easy to keep the old functions while making it easy for people to choose the newer ones, and migrate code if desired. There would then be no need to get rid of the old functions and break backward compatibility. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From henrik.x.nord@REDACTED Tue Nov 28 08:32:15 2017 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Tue, 28 Nov 2017 07:32:15 +0000 Subject: [erlang-questions] Contributor License Agreement for pullrequests In-Reply-To: <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> References: <1511778407.3773.12.camel@ericsson.com> <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> Message-ID: <1511854334.3773.27.camel@ericsson.com> On m?n, 2017-11-27 at 18:06 +0100, Anthony Ramine wrote: > What's the rationale for this? Same as before. To make sure that the code is free to use for the community and us. That you ensure that it is your code to submit. And that you agree to the licence in question for the code you submit to Erlang/OTP. (Apache License 2.0) Before there was a email passed to and fro and the automatic handeling for that was deptricated when we started using github as a master repo, and pullrequests more frequently. This should make the process smoother for everyone. /Henrik > > > > Le 27 nov. 2017 ? 11:26, Henrik Nord X > > a ?crit : > > > > Hello Erlang Questions! > > > > > > As some of you already noticed, we have now added a CLA application > > for > > github pullrequests. > > > > This is done to ensure that we (Erlang/OTP at Ericsson) always have > > a > > up to date list of contributors, and that you contribute code that > > you > > are yours to commit, and free for us to use and maintain under > > the Apache License 2.0. > > > > If and When you submit a pullrequest for the first time (after this > > change) you will be prompted via e-mail and a comment on your > > pullrequest to visit the https://cla-assistant.io/ and agree to > > this. > > > > We will then store your github username, first and last name and > > email > > in much the same way as before. > > > > BR > > Henrik Nord > > Erlang/OTP > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions From list1@REDACTED Tue Nov 28 09:16:15 2017 From: list1@REDACTED (Grzegorz Junka) Date: Tue, 28 Nov 2017 08:16:15 +0000 Subject: [erlang-questions] Wrap my mind around NoSQL (Mnesia) In-Reply-To: <20171124145038.GA23275@auron> References: <20171124145038.GA23275@auron> Message-ID: <9f1b3cd8-386c-839b-da67-89a96d92c9e5@gjunka.com> Hi, You need to consider the impact that storing a list of user tags or urls in one filed can have on query times, i.e. {username, {urls, [tags]}} would require that you read and store the complete list of urls and tags when you need to add or remove one element. I would rather have a table with records {userId, url}. It will automatically have an index on userId. And then I would construct a select to query all urls for the given userId. Then a separate table {userName, userId}. Then a separate table {{userId, url}, tag} where {userId, url} is the key. I could also have {url, urlId} since multiple users can store the same url. That would save memory on having to remember the same url string multiple times. Then for any user I would have {userId, urlId} and {{userId, urlId}, tag}. I hope that helps. GrzegorzJ On 24/11/2017 14:50, Silas wrote: > Hi! > > (This is mainly a question about NoSQL conceptions.? If it is the > wrong place to ask, sorry for that and, please, let me know). > > After years using RDBMS (with structured and OO paradigms at the > programming language side) I'm diving into functional programming.? > When choosing a programming language to practice, I immediately fell > in love with Erlang for this simplicity and beauty. > > After reading some documentation, some chapters of a book and trying > simple projects I decided to try a more complex project with Mnesia.? > It is a social bookmarks website (something similar to what > del.icio.us is) where user can bookmark a URL and associate one or > more tags with it.? For instance, some could bookmark "erlang.org" to > tags "functionalprogramming", "programminglanguage" and others.? Users > also can query other users bookmarks, see what links are associated > with a tag and what tags are associated with a given bookmark. > > In my mind, an RDBMS is a perfect solution for this problem.? I'm > trying to fit it into the key -> value model, though.? My first > question is: is this possible or I'm just doing the wrong thing by > forcing the wrong solution? > > For what I understood, I need to get rid of normalization rules. So I > devised something more or less like that: > > User table: {username, {urls, [tags]}} % set > UserTag table: {user, [tags]} % set > TagUrls table: {tag, url} % bag > TagUsers table: {tag, user} % bag > Urlusers table: {url, user} %bag > UrlTags table: {url, tag} %bag > > It is still a bit confusing where I'd use a set, a bag or an > ordered_set but I think you got the whole picture. > > Any help? > > Thanks! > From tuncer.ayaz@REDACTED Tue Nov 28 11:57:57 2017 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 28 Nov 2017 10:57:57 +0000 Subject: [erlang-questions] Contributor License Agreement for pullrequests In-Reply-To: <1511854334.3773.27.camel@ericsson.com> References: <1511778407.3773.12.camel@ericsson.com> <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> <1511854334.3773.27.camel@ericsson.com> Message-ID: On 11/28/17, Henrik Nord X wrote: > To make sure that the code is free to use for the community and us. > That you ensure that it is your code to submit. > And that you agree to the licence in question for the code you submit > to Erlang/OTP. (Apache License 2.0) So it's just a DCO (Developer's Certificate of Origin)[1] and not a CLA. Correct? [1] https://eclipse.org/legal/DCO.php From tuncer.ayaz@REDACTED Tue Nov 28 12:02:18 2017 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 28 Nov 2017 11:02:18 +0000 Subject: [erlang-questions] Contributor License Agreement for pullrequests In-Reply-To: References: <1511778407.3773.12.camel@ericsson.com> <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> <1511854334.3773.27.camel@ericsson.com> Message-ID: On 11/28/17, Tuncer Ayaz wrote: > On 11/28/17, Henrik Nord X wrote: > > > To make sure that the code is free to use for the community and us. > > That you ensure that it is your code to submit. > > And that you agree to the licence in question for the code you > > submit to Erlang/OTP. (Apache License 2.0) > > So it's just a DCO (Developer's Certificate of Origin)[1] and not a CLA. > > Correct? Sorry, forgot: If it's a DCO, then "Contributor License Agreement" is an unfortunate email subject, since that suggests there might be copyright assignment. From carlsson.richard@REDACTED Tue Nov 28 15:19:23 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 28 Nov 2017 15:19:23 +0100 Subject: [erlang-questions] Contributor License Agreement for pullrequests In-Reply-To: References: <1511778407.3773.12.camel@ericsson.com> <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> <1511854334.3773.27.camel@ericsson.com> Message-ID: There's not a huge difference as far as I can see. In particular, I don't see that a CLA necessarily suggests any copyright assignment, even if this is sometimes done. https://en.wikipedia.org/wiki/Contributor_License_Agreement This page gives a good comparison: https://julien.ponge.org/blog/developer-certificate-of-origin-versus-contributor-license-agreements/ /Richard 2017-11-28 12:02 GMT+01:00 Tuncer Ayaz : > On 11/28/17, Tuncer Ayaz wrote: > > On 11/28/17, Henrik Nord X wrote: > > > > > To make sure that the code is free to use for the community and us. > > > That you ensure that it is your code to submit. > > > And that you agree to the licence in question for the code you > > > submit to Erlang/OTP. (Apache License 2.0) > > > > So it's just a DCO (Developer's Certificate of Origin)[1] and not a CLA. > > > > Correct? > > Sorry, forgot: > > If it's a DCO, then "Contributor License Agreement" is an unfortunate > email subject, since that suggests there might be copyright > assignment. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik.x.nord@REDACTED Tue Nov 28 15:30:07 2017 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Tue, 28 Nov 2017 14:30:07 +0000 Subject: [erlang-questions] Contributor License Agreement for pullrequests In-Reply-To: References: <1511778407.3773.12.camel@ericsson.com> <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> <1511854334.3773.27.camel@ericsson.com> Message-ID: <1511879406.3773.45.camel@ericsson.com> Yea, the thought is that you agree to this anyways when sending in a pullrequest.? The license information is in (almost)all file headers, and a copy is located in the repo itself.? if it should or should not be CLA or DCO? Our thought was that this should be the least bothersome alternative for you all. While still giving us a tool in the case of a copyright dispute. If you rather have a sign-off on all commits I think we are willing to pursue such a option as well. On tis, 2017-11-28 at 15:19 +0100, Richard Carlsson wrote: > There's not a huge difference as far as I can see. In particular, I > don't see that a CLA necessarily suggests any copyright assignment, > even if this is sometimes done.?https://en.wikipedia.org/wiki/Contrib > utor_License_Agreement > > This page gives a good comparison:?https://julien.ponge.org/blog/deve > loper-certificate-of-origin-versus-contributor-license-agreements/ > > ? ? ? ? /Richard > > 2017-11-28 12:02 GMT+01:00 Tuncer Ayaz : > > On 11/28/17, Tuncer Ayaz wrote: > > > On 11/28/17, Henrik Nord X wrote: > > > > > > > To make sure that the code is free to use for the community and > > us. > > > > That you ensure that it is your code to submit. > > > > And that you agree to the licence in question for the code you > > > > submit to Erlang/OTP. (Apache License 2.0) > > > > > > So it's just a DCO (Developer's Certificate of Origin)[1] and not > > a CLA. > > > > > > Correct? > > > > Sorry, forgot: > > > > If it's a DCO, then "Contributor License Agreement" is an > > unfortunate > > email subject, since that suggests there might be copyright > > assignment. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > From tuncer.ayaz@REDACTED Tue Nov 28 16:10:50 2017 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 28 Nov 2017 15:10:50 +0000 Subject: [erlang-questions] Contributor License Agreement for pullrequests In-Reply-To: References: <1511778407.3773.12.camel@ericsson.com> <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> <1511854334.3773.27.camel@ericsson.com> Message-ID: On 11/28/17, Richard Carlsson wrote: > There's not a huge difference as far as I can see. In particular, I > don't see that a CLA necessarily suggests any copyright assignment, > even if this is sometimes done. Right, which is why I wrote "might". A less ambiguous wording would be: "since some CLAs require copyright assignment". Still, given that prominent CLAs have included a copyright assignment clause, it's the first thing many contributors will wonder about when they encounter a CLA requirement. Thus, I think that ditching the name CLA in favor of DCO will communicate the agreement without generating doubt or discouraging some contributors who see "CLA" and shy away. From tuncer.ayaz@REDACTED Tue Nov 28 16:27:29 2017 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 28 Nov 2017 15:27:29 +0000 Subject: [erlang-questions] Contributor License Agreement for pullrequests In-Reply-To: <1511879406.3773.45.camel@ericsson.com> References: <1511778407.3773.12.camel@ericsson.com> <821E85B9-32EF-4E28-B7E6-62F1CD99B1D2@gmail.com> <1511854334.3773.27.camel@ericsson.com> <1511879406.3773.45.camel@ericsson.com> Message-ID: On 11/28/17, Henrik Nord X wrote: > Yea, the thought is that you agree to this anyways when sending in a > pullrequest. > The license information is in (almost)all file headers, and a copy > is located in the repo itself. True and since the SCO legal drama we have "commit --signoff" to make it explicit. > if it should or should not be CLA or DCO? > Our thought was that this should be the least bothersome alternative > for you all. While still giving us a tool in the case of a copyright > dispute. Your thought is valid, the name can be improved. I don't think anyone will object to a DCO and that Anthony asking for the rationale is a sign that CLA is a loaded term historically, causing uncertainty. Just to be clear, CLA which is just a DCO won't raise eyebrows, and naming it DCO makes it unambiguous while also avoiding doubt or contributor hesitation. Yes it's stupid, but names acquire meanings that become hard to clean off. Humans are silly like that. > If you rather have a sign-off on all commits I think we are willing > to pursue such a option as well. I'm not qualified to comment on the legal difference between "git commit --signoff" vs a digitally signed DCO on record, available to Ericsson, and if "--signoff" is a sufficient replacement. From lloyd@REDACTED Tue Nov 28 16:36:26 2017 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Tue, 28 Nov 2017 10:36:26 -0500 Subject: [erlang-questions] Strings - deprecated functions In-Reply-To: <1e3ae868-5b0f-e99d-c672-784164d1cb45@ninenines.eu> References: <7326453.QddQBcBGA1@changa> <0a4f11fd-bdd3-f70b-c165-81f94314d47b@ninenines.eu> <3ac3be52-a69b-054e-a7fc-dca152639146@ninenines.eu> <7A89A617-498E-4423-9590-B491B31BBB34@gmail.com> <1e3ae868-5b0f-e99d-c672-784164d1cb45@ninenines.eu> Message-ID: <2E6AFFB8-8851-4A4B-8133-F882B6C733D6@writersglen.com> Hello, +1 re: the ordering of modules and functions in stdlib. I'm fine with alphabetical ordering. But I do feel that complementary schemes are worth consideration. So here's a challenge: Suppose we have a parallel ordering of modules and functions within modules by frequency of use in the wild. Shouldn't be too hard to generate with a few well-chosen NLP tools. I'd do it myself if I could spare a Saturday morning. But I can't. So I'll offer a bounty--- a $100.00 USD Amazon gift card to first person who generates such a list based on a statistical representative sample of all Erlang source code posted on GitHub within the next month. All the best, LRP > On Nov 27, 2017, at 4:22 PM, Lo?c Hoguin wrote: > > The reason this PR was closed is a little ridiculous. > > stdlib has erl_internal and other modules before most modules that are useful to most people and the important modules are effectively hidden in this confusing mess of useful and not-so-useful-at-all modules. > > Meanwhile the compiler application has exactly one module right now; you're not going to confuse anyone by adding a couple more modules. > > Ridiculous. > >> On 11/27/2017 06:01 PM, Anthony Ramine wrote: >> I don't particularly care about the old functions myself (if you want to run old code, just run a damn old VM, problem solved), but a way to reorganise the documentation to accommodate for that would also bring closure to https://github.com/erlang/otp/pull/363, which would be nice. >>> Le 27 nov. 2017 ? 03:03, Robert Virding a ?crit : >>> >>> I think it is perfectly possible to keep the old functions in the same 'string' module in a clean way if you just reorganise the documentation. Now all the functions, both the new and the obsolete, are kept in one alphabetical list which makes it difficult to see which are which and get a clear overview of the functions. Why not just separate the functions into 2 groups, the new and the obsolete, each in its own alphabetical list? This would make it easy to keep the old functions while making it easy for people to choose the newer ones, and migrate code if desired. There would then be no need to get rid of the old functions and break backward compatibility. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From keam7095@REDACTED Tue Nov 28 21:30:56 2017 From: keam7095@REDACTED (Hakim Fajardo) Date: Tue, 28 Nov 2017 15:30:56 -0500 Subject: [erlang-questions] Examples of erlang-otp used for ecommerce development Message-ID: Hi, I'm interested in learning more about erlang-otp, elixir & phoenix used as an ecommerce solution (website and mobile). Does anyone know any resources available? Are there any examples out there now? Thank you, Hakim -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickard@REDACTED Thu Nov 30 12:51:00 2017 From: rickard@REDACTED (Rickard Green) Date: Thu, 30 Nov 2017 12:51:00 +0100 Subject: [erlang-questions] Erlang Language Specification Message-ID: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> Hi, In the late 90ies work were being done on an Erlang language specification. A draft which was produced can be found at . Unfortunately the work on this document stopped after this. This document is of course quite outdated, but it contains lots of valuable information. I think it is about time to continue this work. It will however require lots of work to finish. We publish the source for the draft in the hope that we in collaboration with the Erlang community eventually will be able to produce an official Erlang Language Specification. I've created a new repository which contains the source of the document. Regards, Rickard Green, Erlang/OTP, Ericsson AB From carlsson.richard@REDACTED Thu Nov 30 14:40:30 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 30 Nov 2017 14:40:30 +0100 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> Message-ID: Great! I've been trying to prod people over the years to dig up these old original files. (I only ever had a copy of the generated .ps/.pdf myself.) Who's been sitting on them? Note that in src/es.tex, lines 22-23, it's better for now if you made STYLE 0 be the default, because it described the actual implementation of Erlang at the time ("4.7.3"), while STYLE 1 "Standard Erlang" described an ideal new standard that had some possibly incompatible changes, several of which were never actually implemented. /Richard 2017-11-30 12:51 GMT+01:00 Rickard Green : > Hi, > > In the late 90ies work were being done on an Erlang language > specification. A draft which was produced can be found at < > http://www.erlang.org/download/erl_spec47.ps.gz>. Unfortunately the work > on this document stopped after this. > > This document is of course quite outdated, but it contains lots of > valuable information. I think it is about time to continue this work. It > will however require lots of work to finish. We publish the source for the > draft in the hope that we in collaboration with the Erlang community > eventually will be able to produce an official Erlang Language > Specification. > > I've created a new repository which > contains the source of the document. > > Regards, > Rickard Green, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Thu Nov 30 14:47:07 2017 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Thu, 30 Nov 2017 14:47:07 +0100 Subject: [erlang-questions] EEP 47 has been slighty updated Message-ID: I have added a note that the stacktrace variable must not be previously bound to the Specification section of the EEP. There are no other changes. https://github.com/bjorng/eep/blob/master/eeps/eep-0047.md /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From rickard@REDACTED Thu Nov 30 15:24:38 2017 From: rickard@REDACTED (Rickard Green) Date: Thu, 30 Nov 2017 15:24:38 +0100 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> Message-ID: On 11/30/2017 02:40 PM, Richard Carlsson wrote: > Great! I've been trying to prod people over the years to dig up these old > original files. (I only ever had a copy of the generated .ps/.pdf myself.) I've unsuccessfully tried to dig them up a couple of times myself over the years. I gave up and began reconstructing the source from the postscript-file, but then all of a sudden the original source landed on my desk :-) > Who's been sitting on them? > Sorry, that is confidential information :-) > Note that in src/es.tex, lines 22-23, it's better for now if you made STYLE > 0 be the default, because it described the actual implementation of Erlang > at the time ("4.7.3"), while STYLE 1 "Standard Erlang" described an ideal > new standard that had some possibly incompatible changes, several of which > were never actually implemented. > Yes, I agree. Regards, Rickard > > /Richard > > 2017-11-30 12:51 GMT+01:00 Rickard Green : > >> Hi, >> >> In the late 90ies work were being done on an Erlang language >> specification. A draft which was produced can be found at < >> http://www.erlang.org/download/erl_spec47.ps.gz>. Unfortunately the work >> on this document stopped after this. >> >> This document is of course quite outdated, but it contains lots of >> valuable information. I think it is about time to continue this work. It >> will however require lots of work to finish. We publish the source for the >> draft in the hope that we in collaboration with the Erlang community >> eventually will be able to produce an official Erlang Language >> Specification. >> >> I've created a new repository which >> contains the source of the document. >> >> Regards, >> Rickard Green, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > From dmkolesnikov@REDACTED Thu Nov 30 16:01:42 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 30 Nov 2017 17:01:42 +0200 Subject: [erlang-questions] Order of monitor signal(s) Message-ID: Hello, There is a statement that Erlang messages may be not delivered, but message order is always preserved relative to two processes. This is a statement regarding the ordinal message? Is this statement valid when we are talking about monitor messages in relation to ordinal message? I have strong feeling that it is not guaranteed. So, let?s assume we have a process A that sends async message to process B and quits. The process B listens messages for process A and monitors it with erlang:monitor(process, A). I am suspecting that there are no guarantee that message will be delivered before exit signal. Am I right? Thank you in advanced! Best Regards, Dmitry From vinoski@REDACTED Thu Nov 30 16:04:11 2017 From: vinoski@REDACTED (Steve Vinoski) Date: Thu, 30 Nov 2017 10:04:11 -0500 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> Message-ID: On Thu, Nov 30, 2017 at 6:51 AM, Rickard Green wrote: > Hi, > > In the late 90ies work were being done on an Erlang language > specification. A draft which was produced can be found at < > http://www.erlang.org/download/erl_spec47.ps.gz>. Unfortunately the work > on this document stopped after this. > > This document is of course quite outdated, but it contains lots of > valuable information. I think it is about time to continue this work. It > will however require lots of work to finish. We publish the source for the > draft in the hope that we in collaboration with the Erlang community > eventually will be able to produce an official Erlang Language > Specification. > > I've created a new repository which > contains the source of the document. > I see the makefiles produce postscript by default, which was useful in the 90s but IMO isn't anymore. Wouldn't it be better to make pdf the default? --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Thu Nov 30 16:05:31 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Thu, 30 Nov 2017 16:05:31 +0100 Subject: [erlang-questions] Order of monitor signal(s) In-Reply-To: References: Message-ID: message delivery is instant the moment you execute send command (send opcode). So exit message well always arrive second, because ordering guarantee - exit message was sent SECOND. On 30 Nov 2017 4:02 pm, "Dmitry Kolesnikov" wrote: Hello, There is a statement that Erlang messages may be not delivered, but message order is always preserved relative to two processes. This is a statement regarding the ordinal message? Is this statement valid when we are talking about monitor messages in relation to ordinal message? I have strong feeling that it is not guaranteed. So, let?s assume we have a process A that sends async message to process B and quits. The process B listens messages for process A and monitors it with erlang:monitor(process, A). I am suspecting that there are no guarantee that message will be delivered before exit signal. Am I right? Thank you in advanced! Best Regards, Dmitry _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Thu Nov 30 16:08:53 2017 From: n.oxyde@REDACTED (Anthony Ramine) Date: Thu, 30 Nov 2017 16:08:53 +0100 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> Message-ID: <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> Would be nice to rewrite that in HTML, we are in 2017 after all. > Le 30 nov. 2017 ? 12:51, Rickard Green a ?crit : > > Hi, > > In the late 90ies work were being done on an Erlang language specification. A draft which was produced can be found at . Unfortunately the work on this document stopped after this. > > This document is of course quite outdated, but it contains lots of valuable information. I think it is about time to continue this work. It will however require lots of work to finish. We publish the source for the draft in the hope that we in collaboration with the Erlang community eventually will be able to produce an official Erlang Language Specification. > > I've created a new repository which contains the source of the document. > > Regards, > Rickard Green, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From rickard@REDACTED Thu Nov 30 16:23:00 2017 From: rickard@REDACTED (Rickard Green) Date: Thu, 30 Nov 2017 16:23:00 +0100 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> Message-ID: On 11/30/2017 04:04 PM, Steve Vinoski wrote: > On Thu, Nov 30, 2017 at 6:51 AM, Rickard Green wrote: > >> Hi, >> >> In the late 90ies work were being done on an Erlang language >> specification. A draft which was produced can be found at < >> http://www.erlang.org/download/erl_spec47.ps.gz>. Unfortunately the work >> on this document stopped after this. >> >> This document is of course quite outdated, but it contains lots of >> valuable information. I think it is about time to continue this work. It >> will however require lots of work to finish. We publish the source for the >> draft in the hope that we in collaboration with the Erlang community >> eventually will be able to produce an official Erlang Language >> Specification. >> >> I've created a new repository which >> contains the source of the document. >> > > I see the makefiles produce postscript by default, which was useful in the > 90s but IMO isn't anymore. Wouldn't it be better to make pdf the default? > > --steve > Yes, I think that would be better too. The current makefile use pdflatex to produce the pdf. I'm not sure if it is as available in latex packages as the latex command is. Since I don't know I went for the ps as default, but this can be changed. Regards, Rickard From sverker.eriksson@REDACTED Thu Nov 30 16:28:59 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 30 Nov 2017 15:28:59 +0000 Subject: [erlang-questions] Order of monitor signal(s) In-Reply-To: References: Message-ID: <1512055738.3545.22.camel@ericsson.com> On tor, 2017-11-30 at 16:05 +0100, Dmytro Lytovchenko wrote: > message delivery is instant the moment you execute send command (send > opcode). NO NO NO. Messages are asynchronous signals. The return of the "send command" does not guarantee ANYTHING. > So exit message well always arrive second, because ordering guarantee > - exit message was sent SECOND. > YES Erlang signals (messages, links, monitors, etc) guarantee ordered delivery between process pairs. THAT is what gives you the guarantee that all messages sent from process A are put in the message queue of process B before the monitor 'DOWN' message. /Sverker > On 30 Nov 2017 4:02 pm, "Dmitry Kolesnikov" > wrote: > Hello, > > There is a statement that Erlang messages may be not delivered, but > message order is always preserved relative to two processes. This is > a statement regarding the ordinal message? Is this statement valid > when we are talking about monitor messages in relation to ordinal > message? I have strong feeling that it is not guaranteed. > > So, let?s assume we have a process A that sends async message to > process B and quits. The process B listens messages for process A and > monitors it with erlang:monitor(process, A). I am suspecting that > there are no guarantee that message will be delivered before exit > signal. Am I right? > > Thank you in advanced! > > Best Regards, > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From rickard@REDACTED Thu Nov 30 16:29:44 2017 From: rickard@REDACTED (Rickard Green) Date: Thu, 30 Nov 2017 16:29:44 +0100 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> Message-ID: On 11/30/2017 04:08 PM, Anthony Ramine wrote: > Would be nice to rewrite that in HTML, we are in 2017 after all. > No I don't think it would be a good ide to rewrite this in HTML. There might be other options than latex, but I don't see that HTML would be an option. This due to the mathematical formulas in the document. There are tools available to produce html from latex which can be integrated in the makefiles which of course would be nice. Regards, Rickard >> Le 30 nov. 2017 ? 12:51, Rickard Green a ?crit : >> >> Hi, >> >> In the late 90ies work were being done on an Erlang language specification. A draft which was produced can be found at . Unfortunately the work on this document stopped after this. >> >> This document is of course quite outdated, but it contains lots of valuable information. I think it is about time to continue this work. It will however require lots of work to finish. We publish the source for the draft in the hope that we in collaboration with the Erlang community eventually will be able to produce an official Erlang Language Specification. >> >> I've created a new repository which contains the source of the document. >> >> Regards, >> Rickard Green, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From S.J.Thompson@REDACTED Thu Nov 30 16:33:20 2017 From: S.J.Thompson@REDACTED (Simon Thompson) Date: Thu, 30 Nov 2017 15:33:20 +0000 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> Message-ID: <0156C2A0-1919-45D0-A693-16EC62D6E80D@kent.ac.uk> Once it?s stable would be great to transform the document into something executable (e.g. Erlang, Haskell) or even verifiable (Isabelle, Coq, K-framework) so that there?s a ?reference implementation? which can be tested and verified. Simon > On 30 Nov 2017, at 15:29, Rickard Green wrote: > > On 11/30/2017 04:08 PM, Anthony Ramine wrote: >> Would be nice to rewrite that in HTML, we are in 2017 after all. > > No I don't think it would be a good ide to rewrite this in HTML. There might be other options than latex, but I don't see that HTML would be an option. This due to the mathematical formulas in the document. > > There are tools available to produce html from latex which can be integrated in the makefiles which of course would be nice. > > Regards, > Rickard > Simon Thompson | Professor of Logic and Computation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Nov 30 16:37:56 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 30 Nov 2017 15:37:56 +0000 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: <0156C2A0-1919-45D0-A693-16EC62D6E80D@kent.ac.uk> References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> <0156C2A0-1919-45D0-A693-16EC62D6E80D@kent.ac.uk> Message-ID: I might be interested in building a Coq/Agda/Twelf model for sure :) But given that, it is often good to have a interpreted model in a more "normal" programming language as well in my experience. They tend to be more tangible in many cases. Perhaps a Coq extraction will do? On Thu, Nov 30, 2017 at 4:33 PM Simon Thompson wrote: > > Once it?s stable would be great to transform the document into something > executable (e.g. Erlang, Haskell) or even verifiable (Isabelle, Coq, > K-framework) so that there?s a ?reference implementation? which can be > tested and verified. > > Simon > > > On 30 Nov 2017, at 15:29, Rickard Green wrote: > > On 11/30/2017 04:08 PM, Anthony Ramine wrote: > > Would be nice to rewrite that in HTML, we are in 2017 after all. > > > No I don't think it would be a good ide to rewrite this in HTML. There > might be other options than latex, but I don't see that HTML would be an > option. This due to the mathematical formulas in the document. > > There are tools available to produce html from latex which can be > integrated in the makefiles which of course would be nice. > > Regards, > Rickard > > > Simon Thompson | Professor of Logic and Computation > School of Computing | University of Kent | Canterbury, CT2 7NF, UK > s.j.thompson@REDACTED | M +44 7986 085754 <+44%207986%20085754> | W > www.cs.kent.ac.uk/~sjt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Thu Nov 30 17:56:19 2017 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 30 Nov 2017 17:56:19 +0100 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: <0156C2A0-1919-45D0-A693-16EC62D6E80D@kent.ac.uk> References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> <0156C2A0-1919-45D0-A693-16EC62D6E80D@kent.ac.uk> Message-ID: On Thu, Nov 30, 2017 at 4:33 PM, Simon Thompson wrote: > > Once it?s stable would be great to transform the document into something > executable (e.g. Erlang, Haskell) or even verifiable (Isabelle, Coq, > K-framework) so that there?s a ?reference implementation? which can be > tested and verified. > > Simon > Great idea - I'd go for an XML input with a custom DTD - this is pretty easy to transform into PDF (via LaTeX or Apache FOP) or HTML and subsets can be extracted for automatic code generation and verification. I've often used XML for this, generating code, manual pages and PDFs from a single input. /Joe > > On 30 Nov 2017, at 15:29, Rickard Green wrote: > > On 11/30/2017 04:08 PM, Anthony Ramine wrote: > > Would be nice to rewrite that in HTML, we are in 2017 after all. > > > No I don't think it would be a good ide to rewrite this in HTML. There might > be other options than latex, but I don't see that HTML would be an option. > This due to the mathematical formulas in the document. > > There are tools available to produce html from latex which can be integrated > in the makefiles which of course would be nice. > > Regards, > Rickard > > > Simon Thompson | Professor of Logic and Computation > School of Computing | University of Kent | Canterbury, CT2 7NF, UK > s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From S.J.Thompson@REDACTED Thu Nov 30 17:58:00 2017 From: S.J.Thompson@REDACTED (Simon Thompson) Date: Thu, 30 Nov 2017 16:58:00 +0000 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> <0156C2A0-1919-45D0-A693-16EC62D6E80D@kent.ac.uk> Message-ID: Yes, I?ve done this, but did trip up with some of the escaping conventions for multiple languages ? > On 30 Nov 2017, at 16:56, Joe Armstrong wrote: > > Great idea - I'd go for an XML input with a custom DTD - this is > pretty easy to transform > into PDF (via LaTeX or Apache FOP) or HTML and subsets can be extracted for > automatic code generation and verification. > > I've often used XML for this, generating code, manual pages and PDFs > from a single > input. > > /Joe > > > Simon Thompson | Professor of Logic and Computation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Thu Nov 30 18:09:01 2017 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 30 Nov 2017 18:09:01 +0100 Subject: [erlang-questions] Order of monitor signal(s) In-Reply-To: <1512055738.3545.22.camel@ericsson.com> References: <1512055738.3545.22.camel@ericsson.com> Message-ID: On Thu, Nov 30, 2017 at 4:28 PM, Sverker Eriksson wrote: > On tor, 2017-11-30 at 16:05 +0100, Dmytro Lytovchenko wrote: >> message delivery is instant the moment you execute send command (send >> opcode). > > NO NO NO. > Messages are asynchronous signals. The return of the "send command" > does not guarantee ANYTHING. > > >> So exit message well always arrive second, because ordering guarantee >> - exit message was sent SECOND. >> > YES > Erlang signals (messages, links, monitors, etc) guarantee ordered > delivery between process pairs. THAT is what gives you the guarantee > that all messages sent from process A are put in the message queue of > process B before the monitor 'DOWN' message. To start with these properties different depending upon the environment. You might have a) pairs of processes in the same node b) pairs of processes on different nodes on the same host and c) pairs of processes on different nodes on different hosts. The failure modes in the three cases all different. I think you should say "in the absence of any failures and counter indications" messages and exit signals between pairs of processes should arrive in the order they were sent. In the distributed case the messages and exits all go down the same socket in order, so they should arrive ordered. If the socket closes abnormally, or ping watchguard timer elapses you will be told. The general idea is that ordering is preserved between pairs of processes if nothing goes wrong, and if something goes wrong you will be told. In order to be told you need to be trapping exits and node up/down messages - the 'something has gone wrong' message will arrive *after* the failure has occurred - it's up to you to figure out what to do. Cheers /Joe > > /Sverker > > >> On 30 Nov 2017 4:02 pm, "Dmitry Kolesnikov" >> wrote: >> Hello, >> >> There is a statement that Erlang messages may be not delivered, but >> message order is always preserved relative to two processes. This is >> a statement regarding the ordinal message? Is this statement valid >> when we are talking about monitor messages in relation to ordinal >> message? I have strong feeling that it is not guaranteed. >> >> So, let?s assume we have a process A that sends async message to >> process B and quits. The process B listens messages for process A and >> monitors it with erlang:monitor(process, A). I am suspecting that >> there are no guarantee that message will be delivered before exit >> signal. Am I right? >> >> Thank you in advanced! >> >> Best Regards, >> Dmitry >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From n.oxyde@REDACTED Thu Nov 30 18:33:58 2017 From: n.oxyde@REDACTED (Anthony Ramine) Date: Thu, 30 Nov 2017 18:33:58 +0100 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> Message-ID: <362DC334-DDD6-460B-9B8A-61811A6FAFD9@gmail.com> The most important part of a spec isn't the verbosity of its few mathematical formulas when editing, it's good cross-reference linking, internally in externally. The format that excels at that is HTML. There are various ways to have mathematical formulas in HTML. > Le 30 nov. 2017 ? 16:29, Rickard Green a ?crit : > > On 11/30/2017 04:08 PM, Anthony Ramine wrote: >> Would be nice to rewrite that in HTML, we are in 2017 after all. > > No I don't think it would be a good ide to rewrite this in HTML. There might be other options than latex, but I don't see that HTML would be an option. This due to the mathematical formulas in the document. > > There are tools available to produce html from latex which can be integrated in the makefiles which of course would be nice. > > Regards, > Rickard > >>> Le 30 nov. 2017 ? 12:51, Rickard Green a ?crit : >>> >>> Hi, >>> >>> In the late 90ies work were being done on an Erlang language specification. A draft which was produced can be found at . Unfortunately the work on this document stopped after this. >>> >>> This document is of course quite outdated, but it contains lots of valuable information. I think it is about time to continue this work. It will however require lots of work to finish. We publish the source for the draft in the hope that we in collaboration with the Erlang community eventually will be able to produce an official Erlang Language Specification. >>> >>> I've created a new repository which contains the source of the document. >>> >>> Regards, >>> Rickard Green, Erlang/OTP, Ericsson AB >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions > > From dmytro.lytovchenko@REDACTED Thu Nov 30 19:35:29 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Thu, 30 Nov 2017 19:35:29 +0100 Subject: [erlang-questions] Order of monitor signal(s) In-Reply-To: <1512055738.3545.22.camel@ericsson.com> References: <1512055738.3545.22.camel@ericsson.com> Message-ID: On 30 Nov 2017 4:29 pm, "Sverker Eriksson" wrote: On tor, 2017-11-30 at 16:05 +0100, Dmytro Lytovchenko wrote: > message delivery is instant the moment you execute send command (send > opcode). NO NO NO. Messages are asynchronous signals. The return of the "send command" does not guarantee ANYTHING. unless send is a trappable bif, it delivers to mail box immediately, this is what i meant to say. of course it will be delivered to the Erlang code when the process wakes up and gets his cpu time. terminology is not easy. > So exit message well always arrive second, because ordering guarantee > - exit message was sent SECOND. > YES Erlang signals (messages, links, monitors, etc) guarantee ordered delivery between process pairs. THAT is what gives you the guarantee that all messages sent from process A are put in the message queue of process B before the monitor 'DOWN' message. /Sverker > On 30 Nov 2017 4:02 pm, "Dmitry Kolesnikov" > wrote: > Hello, > > There is a statement that Erlang messages may be not delivered, but > message order is always preserved relative to two processes. This is > a statement regarding the ordinal message? Is this statement valid > when we are talking about monitor messages in relation to ordinal > message? I have strong feeling that it is not guaranteed. > > So, let?s assume we have a process A that sends async message to > process B and quits. The process B listens messages for process A and > monitors it with erlang:monitor(process, A). I am suspecting that > there are no guarantee that message will be delivered before exit > signal. Am I right? > > Thank you in advanced! > > Best Regards, > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Thu Nov 30 21:09:25 2017 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 30 Nov 2017 22:09:25 +0200 Subject: [erlang-questions] Order of monitor signal(s) In-Reply-To: References: <1512055738.3545.22.camel@ericsson.com> Message-ID: <20A24D5A-08DF-4E0F-BE62-6B5F964E36E3@gmail.com> Hello, Thank you for your feedback and good point about the environment! Yes, I am talking about pairs of processes in the same node. Do you know if similar ordering rules applies for port messages? Can it send {exit_status, ?} before {data, ?}? It seems that I am experience this situation randomly at OTP 20.0 when port is opened to bash script with following parameters: binary, stream, exit_status, use_stdio, stderr_to_stdout Best Regards, Dmitry > On 30 Nov 2017, at 19.09, Joe Armstrong wrote: > > On Thu, Nov 30, 2017 at 4:28 PM, Sverker Eriksson > > wrote: >> On tor, 2017-11-30 at 16:05 +0100, Dmytro Lytovchenko wrote: >>> message delivery is instant the moment you execute send command (send >>> opcode). >> >> NO NO NO. >> Messages are asynchronous signals. The return of the "send command" >> does not guarantee ANYTHING. >> >> >>> So exit message well always arrive second, because ordering guarantee >>> - exit message was sent SECOND. >>> >> YES >> Erlang signals (messages, links, monitors, etc) guarantee ordered >> delivery between process pairs. THAT is what gives you the guarantee >> that all messages sent from process A are put in the message queue of >> process B before the monitor 'DOWN' message. > > To start with these properties different depending upon the environment. > You might have a) pairs of processes in the same node > b) pairs of processes on different nodes on the same host and c) pairs of > processes on different nodes on different hosts. > > The failure modes in the three cases all different. > > I think you should say "in the absence of any failures and counter indications" > messages and exit signals between pairs of processes should arrive in > the order they were sent. > > In the distributed case the messages and exits all go down the same > socket in order, > so they should arrive ordered. If the socket closes abnormally, or > ping watchguard timer > elapses you will be told. > > The general idea is that ordering is preserved between pairs of > processes if nothing goes wrong, > and if something goes wrong you will be told. > > In order to be told you need to be trapping exits and > node up/down messages - the 'something has gone wrong' message will > arrive *after* the > failure has occurred - it's up to you to figure out what to do. > > Cheers > > /Joe > > > >> >> /Sverker >> >> >>> On 30 Nov 2017 4:02 pm, "Dmitry Kolesnikov" >>> wrote: >>> Hello, >>> >>> There is a statement that Erlang messages may be not delivered, but >>> message order is always preserved relative to two processes. This is >>> a statement regarding the ordinal message? Is this statement valid >>> when we are talking about monitor messages in relation to ordinal >>> message? I have strong feeling that it is not guaranteed. >>> >>> So, let?s assume we have a process A that sends async message to >>> process B and quits. The process B listens messages for process A and >>> monitors it with erlang:monitor(process, A). I am suspecting that >>> there are no guarantee that message will be delivered before exit >>> signal. Am I right? >>> >>> Thank you in advanced! >>> >>> Best Regards, >>> Dmitry >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From denc716@REDACTED Thu Nov 30 21:15:09 2017 From: denc716@REDACTED (derek) Date: Thu, 30 Nov 2017 12:15:09 -0800 Subject: [erlang-questions] Patch Package: OTP 19.3.6.4 In-Reply-To: <0a953d81-6984-e875-211f-990dd3dd7495@ericsson.com> References: <0a953d81-6984-e875-211f-990dd3dd7495@ericsson.com> Message-ID: To Ingela and if sverker also on the list: This morning I just found OTP-19.2.3.1 release is available, for the only changed application is erts-8.2.2.1; I'm trying to understand why is this needed? when you already have OTP 19.3.6.3 which included erts-8.3.5.3; how many users are there sticking with 19.2 and can upgrade 19.2.X.Y.latest but can't upgrade to 19.3.X.Y.latest ? as a downstream OS distribution packager, I want to know if this need a packaging as well Could you also talk on what are the Erlang OTP release rules? if not have recorded on some link? and what's the maintenance plan for each past release? how long will each release be supported? like 19 will be supported till Month-Year? 18 till what Month-Year? and is 17 already no longer maintained? https://github.com/erlang/otp/releases https://github.com/erlang/otp/releases/tag/OTP-19.2.3.1 === OTP-19.2.3.1 === sverker tagged this a day ago ? 4766 commits to master since this tag Changed Applications: - erts-8.2.2.1 On Thu, Nov 23, 2017 at 7:26 AM, Ingela Anderton Andin wrote: > Patch Package: OTP 19.3.6.4 > Git Tag: OTP-19.3.6.4 > Date: 2017-11-23 > Trouble Report Id: OTP-14748 > Seq num: > System: OTP > Release: 19 > Application: ssl-8.1.3.1 > Predecessor: OTP 19.3.6.3 > > Check out the git tag OTP-19.3.6.4, and build a full OTP system > including documentation. Apply one or more applications from this > build as patches to your installation using the 'otp_patch_apply' > tool. For information on install requirements, see descriptions for > each application version below. > > --------------------------------------------------------------------- > --- ssl-8.1.3.1 ----------------------------------------------------- > --------------------------------------------------------------------- > > Note! The ssl-8.1.3.1 application can *not* be applied independently > of other applications on an arbitrary OTP 19 installation. > > On a full OTP 19 installation, also the following runtime > dependency has to be satisfied: > -- stdlib-3.2 (first satisfied in OTP 19.2) > > > --- Fixed Bugs and Malfunctions --- > > OTP-14748 Application(s): ssl > > An erlang TLS server configured with cipher suites > using rsa key exchange, may be vulnerable to an > Adaptive Chosen Ciphertext attack (AKA Bleichenbacher > attack) against RSA, which when exploited, may result > in plaintext recovery of encrypted messages and/or a > Man-in-the-middle (MiTM) attack, despite the attacker > not having gained access to the server?s private key > itself. CVE-2017-1000385 > > Exploiting this vulnerability to perform plaintext > recovery of encrypted messages will, in most practical > cases, allow an attacker to read the plaintext only > after the session has completed. Only TLS sessions > established using RSA key exchange are vulnerable to > this attack. > > Exploiting this vulnerability to conduct a MiTM attack > requires the attacker to complete the initial attack, > which may require thousands of server requests, during > the handshake phase of the targeted session within the > window of the configured handshake timeout. This attack > may be conducted against any TLS session using RSA > signatures, but only if cipher suites using RSA key > exchange are also enabled on the server. The limited > window of opportunity, limitations in bandwidth, and > latency make this attack significantly more difficult > to execute. > > RSA key exchange is enabled by default although least > prioritized if server order is honored. For such a > cipher suite to be chosen it must also be supported by > the client and probably the only shared cipher suite. > > Captured TLS sessions encrypted with ephemeral cipher > suites (DHE or ECDHE) are not at risk for subsequent > decryption due to this vulnerability. > > As a workaround if default cipher suite configuration > was used you can configure the server to not use > vulnerable suites with the ciphers option like this: > > {ciphers, [Suite || Suite <- ssl:cipher_suites(), > element(1,Suite) =/= rsa]} > > that is your code will look somethingh like this: > > ssl:listen(Port, [{ciphers, [Suite || Suite <- > ssl:cipher_suites(), element(1,S) =/= rsa]} | > Options]). > > Thanks to Hanno B?ck, Juraj Somorovsky and Craig Young > for reporting this vulnerability. > > > Full runtime dependencies of ssl-8.1.3.1: crypto-3.3, erts-7.0, > inets-5.10.7, kernel-3.0, public_key-1.2, stdlib-3.2 > > > --------------------------------------------------------------------- > --------------------------------------------------------------------- > --------------------------------------------------------------------- > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From kenneth@REDACTED Thu Nov 30 21:49:06 2017 From: kenneth@REDACTED (Kenneth Lundin) Date: Thu, 30 Nov 2017 21:49:06 +0100 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: <362DC334-DDD6-460B-9B8A-61811A6FAFD9@gmail.com> References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> <362DC334-DDD6-460B-9B8A-61811A6FAFD9@gmail.com> Message-ID: There is one thing with having the spec available as html, which would be very useful. But I don't think the source should be html. The guiding principle should be 1 source many possible presentations. /Kenneth Erlang/OTP, Ericsson Den 30 nov. 2017 18:34 skrev "Anthony Ramine" : The most important part of a spec isn't the verbosity of its few mathematical formulas when editing, it's good cross-reference linking, internally in externally. The format that excels at that is HTML. There are various ways to have mathematical formulas in HTML. > Le 30 nov. 2017 ? 16:29, Rickard Green a ?crit : > > On 11/30/2017 04:08 PM, Anthony Ramine wrote: >> Would be nice to rewrite that in HTML, we are in 2017 after all. > > No I don't think it would be a good ide to rewrite this in HTML. There might be other options than latex, but I don't see that HTML would be an option. This due to the mathematical formulas in the document. > > There are tools available to produce html from latex which can be integrated in the makefiles which of course would be nice. > > Regards, > Rickard > >>> Le 30 nov. 2017 ? 12:51, Rickard Green a ?crit : >>> >>> Hi, >>> >>> In the late 90ies work were being done on an Erlang language specification. A draft which was produced can be found at < http://www.erlang.org/download/erl_spec47.ps.gz>. Unfortunately the work on this document stopped after this. >>> >>> This document is of course quite outdated, but it contains lots of valuable information. I think it is about time to continue this work. It will however require lots of work to finish. We publish the source for the draft in the hope that we in collaboration with the Erlang community eventually will be able to produce an official Erlang Language Specification. >>> >>> I've created a new repository which contains the source of the document. >>> >>> Regards, >>> Rickard Green, Erlang/OTP, Ericsson AB >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Thu Nov 30 22:06:26 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 30 Nov 2017 21:06:26 +0000 Subject: [erlang-questions] Patch Package: OTP 19.3.6.4 In-Reply-To: References: <0a953d81-6984-e875-211f-990dd3dd7495@ericsson.com> Message-ID: <1512075986.3545.37.camel@ericsson.com> All announced patches on 19 will have versions OTP-19.3.6.x Any other pushed OTP-19 tag is some sort of special customer request of backporting fix or feature. If something is fixed for example on OTP-19.2.y.z, then it will also be fixed or already have been fixed on OTP-19.3.6.x. /Sverker On tor, 2017-11-30 at 12:15 -0800, derek wrote: > To Ingela and if sverker also on the list: > > This morning I just found OTP-19.2.3.1 release is available, for the > only changed application is erts-8.2.2.1;??I'm trying to understand > why is this needed? when you already have OTP 19.3.6.3 which included > erts-8.3.5.3; > how many users are there sticking with 19.2 and can upgrade > 19.2.X.Y.latest but can't upgrade to 19.3.X.Y.latest??? > as a downstream OS distribution packager, I want to know if this need > a packaging as well > > Could you also talk on what are the Erlang OTP release rules? if not > have recorded on some link? and what's the maintenance plan for each > past release? how long will each release be supported? like 19 will > be > supported till Month-Year? 18 till what Month-Year????and is 17 > already no longer maintained? > > > https://github.com/erlang/otp/releases > https://github.com/erlang/otp/releases/tag/OTP-19.2.3.1 > > === OTP-19.2.3.1 === > > ?sverker tagged this a day ago ? 4766 commits to master since this > tag > > Changed Applications: > - erts-8.2.2.1 > > On Thu, Nov 23, 2017 at 7:26 AM, Ingela Anderton Andin > wrote: > > > > Patch Package:????OTP 19.3.6.4 > > Git Tag:????????????????OTP-19.3.6.4 > > Date:????????????????????2017-11-23 > > Trouble Report Id:??OTP-14748 > > Seq num: > > System:??????????????????OTP > > Release:?????????????????19 > > Application:????????????ssl-8.1.3.1 > > Predecessor:??????????OTP 19.3.6.3 > > > > ?Check out the git tag OTP-19.3.6.4, and build a full OTP system > > ?including documentation. Apply one or more applications from this > > ?build as patches to your installation using the 'otp_patch_apply' > > ?tool. For information on install requirements, see descriptions > > for > > ?each application version below. > > > > ?---------------------------------------------------------------- > > ----- > > ?--- ssl-8.1.3.1 -------------------------------------------------- > > --- > > ?---------------------------------------------------------------- > > ----- > > > > ?Note! The ssl-8.1.3.1 application can *not* be applied > > independently > > ???????of other applications on an arbitrary OTP 19 installation. > > > > ???????On a full OTP 19 installation, also the following runtime > > ???????dependency has to be satisfied: > > ???????-- stdlib-3.2 (first satisfied in OTP 19.2) > > > > > > ?--- Fixed Bugs and Malfunctions --- > > > > ? OTP-14748????Application(s): ssl > > > > ???????????????An erlang TLS server configured with cipher suites > > ???????????????using rsa key exchange, may be vulnerable to an > > ???????????????Adaptive Chosen Ciphertext attack (AKA > > Bleichenbacher > > ???????????????attack) against RSA, which when exploited, may > > result > > ???????????????in plaintext recovery of encrypted messages and/or a > > ???????????????Man-in-the-middle (MiTM) attack, despite the > > attacker > > ???????????????not having gained access to the server?s private key > > ???????????????itself. CVE-2017-1000385 > > > > ???????????????Exploiting this vulnerability to perform plaintext > > ???????????????recovery of encrypted messages will, in most > > practical > > ???????????????cases, allow an attacker to read the plaintext only > > ???????????????after the session has completed. Only TLS sessions > > ???????????????established using RSA key exchange are vulnerable to > > ???????????????this attack. > > > > ???????????????Exploiting this vulnerability to conduct a MiTM > > attack > > ???????????????requires the attacker to complete the initial > > attack, > > ???????????????which may require thousands of server requests, > > during > > ???????????????the handshake phase of the targeted session within > > the > > ???????????????window of the configured handshake timeout. This > > attack > > ???????????????may be conducted against any TLS session using RSA > > ???????????????signatures, but only if cipher suites using RSA key > > ???????????????exchange are also enabled on the server. The limited > > ???????????????window of opportunity, limitations in bandwidth, and > > ???????????????latency make this attack significantly more > > difficult > > ???????????????to execute. > > > > ???????????????RSA key exchange is enabled by default although > > least > > ???????????????prioritized if server order is honored. For such a > > ???????????????cipher suite to be chosen it must also be supported > > by > > ???????????????the client and probably the only shared cipher > > suite. > > > > ???????????????Captured TLS sessions encrypted with ephemeral > > cipher > > ???????????????suites (DHE or ECDHE) are not at risk for subsequent > > ???????????????decryption due to this vulnerability. > > > > ???????????????As a workaround if default cipher suite > > configuration > > ???????????????was used you can configure the server to not use > > ???????????????vulnerable suites with the ciphers option like this: > > > > ???????????????{ciphers, [Suite || Suite <- ssl:cipher_suites(), > > ???????????????element(1,Suite) =/= rsa]} > > > > ???????????????that is your code will look somethingh like this: > > > > ???????????????ssl:listen(Port, [{ciphers, [Suite || Suite <- > > ???????????????ssl:cipher_suites(), element(1,S) =/= rsa]} | > > ???????????????Options]). > > > > ???????????????Thanks to Hanno B?ck, Juraj Somorovsky and Craig > > Young > > ???????????????for reporting this vulnerability. > > > > > > ?Full runtime dependencies of ssl-8.1.3.1: crypto-3.3, erts-7.0, > > ?inets-5.10.7, kernel-3.0, public_key-1.2, stdlib-3.2 > > > > > > ?---------------------------------------------------------------- > > ----- > > ?---------------------------------------------------------------- > > ----- > > ?---------------------------------------------------------------- > > ----- > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Thu Nov 30 23:08:54 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 1 Dec 2017 11:08:54 +1300 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: <362DC334-DDD6-460B-9B8A-61811A6FAFD9@gmail.com> References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> <362DC334-DDD6-460B-9B8A-61811A6FAFD9@gmail.com> Message-ID: On 1/12/17 6:33 AM, Anthony Ramine wrote: > The most important part of a spec isn't the verbosity of its few mathematical formulas when editing, it's good cross-reference linking, internally in externally. The format that excels at that is HTML. There are various ways to have mathematical formulas in HTML. I agree that good cross-referencing is very important. The cross-referencing you can get with good literate programming tools is *awesome*. Look at the TeXBook for an example, or the book about lcc https://www.amazon.com/Retargetable-Compiler-Design-Implementation/dp/0805316701 I've seen four methods used to get mathematical formulas into HTML: - format using (La)TeX and paste in as an image (yuck) - use MathML (I have made repeated attempts to do this myself. Much struggle for poor results.) - use some fancy WYSIWYG tool and have it generate something (WYSINWIS : what you see is not what I see) - use MathJax this seems to work pretty well. It reminds me of Professor CleverByte's Visit to Heaven... Given that PDF supports intra- and inter-document links and text search, I don't see an advantage for HTML here. From ok@REDACTED Thu Nov 30 23:17:50 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 1 Dec 2017 11:17:50 +1300 Subject: [erlang-questions] Erlang Language Specification In-Reply-To: References: <6e411f7e-fb31-539c-3ecd-9e904b942aa7@erlang.org> <93F6768D-EA66-483D-8E54-E6D7D18775DD@gmail.com> <0156C2A0-1919-45D0-A693-16EC62D6E80D@kent.ac.uk> Message-ID: <6f4976c2-8a19-97c5-097e-489145926149@cs.otago.ac.nz> On 1/12/17 5:58 AM, Simon Thompson wrote: > Yes, I?ve done this, but did trip up with some of the escaping > conventions for multiple languages ? One approach goes like this. You have two DTDs. One includes this: So you do things like You have a program that reads the parsed form of the file in either a SAX-like or DOM-like manner, and processes the elements, generating whatever markup is appropriate, and emits the complete result as XML, ready for formatting. *Manual* escaping is tedious and error-prone.