From jesper.louis.andersen@REDACTED Sun Jan 1 14:42:34 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 01 Jan 2017 13:42:34 +0000 Subject: [erlang-questions] Limit the Network Bandwidth per application In-Reply-To: References: Message-ID: See https://github.com/jlouis/rlimit but mind you, there is an issue as well: https://github.com/jlouis/rlimit/issues/2 :) On Sat, Dec 31, 2016 at 9:57 AM Frank Muller wrote: > Hi guys, > > I stumbled across trickle (https://github.com/mariusae/trickle), a > user-land application bandwidth shaper. > A very useful small piece of software as explained here: > > http://www.tecmint.com/manage-and-limit-downloadupload-bandwidth-with-trickle-in-linux/ > > It relies on libevent (http://libevent.org/), and can be installed vi > apt-get or yum. > > I?m wondering if one can create a trickle equivalent in Erlang (+NIF or > whatever)? > > Feedbacks very appreciated (links to equivalent apps in Erlang, > implementation ideas, etc). > > Thank you and happy new year. > /Frank > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sun Jan 1 17:29:59 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sun, 01 Jan 2017 16:29:59 +0000 Subject: [erlang-questions] Limit the Network Bandwidth per application In-Reply-To: References: Message-ID: Thank you for the link Jesper. Actually, any TCP bandwidth limiting (upload, download) code will make it. I would like to learn how to implement such features in practice. /Frank Le dim. 1 janv. 2017 ? 14:42, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> a ?crit : > See https://github.com/jlouis/rlimit but mind you, there is an issue as > well: https://github.com/jlouis/rlimit/issues/2 :) > > On Sat, Dec 31, 2016 at 9:57 AM Frank Muller > wrote: > > Hi guys, > > I stumbled across trickle (https://github.com/mariusae/trickle), a > user-land application bandwidth shaper. > A very useful small piece of software as explained here: > > http://www.tecmint.com/manage-and-limit-downloadupload-bandwidth-with-trickle-in-linux/ > > It relies on libevent (http://libevent.org/), and can be installed vi > apt-get or yum. > > I?m wondering if one can create a trickle equivalent in Erlang (+NIF or > whatever)? > > Feedbacks very appreciated (links to equivalent apps in Erlang, > implementation ideas, etc). > > Thank you and happy new year. > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vans_163@REDACTED Sun Jan 1 18:17:24 2017 From: vans_163@REDACTED (Vans S) Date: Sun, 1 Jan 2017 17:17:24 +0000 (UTC) Subject: [erlang-questions] Limit the Network Bandwidth per application In-Reply-To: References: Message-ID: <1854268120.4982109.1483291044993@mail.yahoo.com> This is more difficult then it appears. The standard way is to use cgroups or window quotas (per user/process). The other way is to throttle at the interface level, https://www.cyberciti.biz/faq/linux-traffic-shaping-using-tc-to-control-http-traffic/. Yet another way is to use virtualization and set quotas on the container/vm. Yet another way is to mark the packets coming from your process so pf/iptables/tc/other can traffic shape it. Probably the most straightforward way is to simply count every call to send/recv in Erlang itself and count the amount of bytes sent/recv then do math. I would probably go for the cgroups or virtualization approach personally. On Sunday, January 1, 2017 11:30 AM, Frank Muller wrote: Thank you for the link Jesper. Actually, any TCP bandwidth limiting (upload, download) code will make it. I would like to learn how to implement such features in practice. /Frank Le dim. 1 janv. 2017 ? 14:42, Jesper Louis Andersen a ?crit : See https://github.com/jlouis/rlimit but mind you, there is an issue as well: https://github.com/jlouis/rlimit/issues/2 :) > >On Sat, Dec 31, 2016 at 9:57 AM Frank Muller wrote: > >Hi guys, >> >>I stumbled across trickle (https://github.com/mariusae/trickle), a user-land application bandwidth shaper. >>A very useful small piece of software as explained here: >>http://www.tecmint.com/manage-and-limit-downloadupload-bandwidth-with-trickle-in-linux/ >> >> >>It relies on libevent (http://libevent.org/), and can be installed vi apt-get or yum. >> >>I?m wondering if one can create a trickle equivalent in Erlang (+NIF or whatever)? >> >>Feedbacks very appreciated (links to equivalent apps in Erlang, implementation ideas, etc). >> >>Thank you and happy new year. >>/Frank > >> >>_______________________________________________ >> >> >>erlang-questions mailing list >> >> >>erlang-questions@REDACTED >> >> >>http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From jay@REDACTED Sun Jan 1 18:37:55 2017 From: jay@REDACTED (Jay Nelson) Date: Sun, 1 Jan 2017 09:37:55 -0800 Subject: [erlang-questions] Limit the Network Bandwidth per application Message-ID: You might want to take a look at https://github.com/duomark/epocxy for cxy_fount. I gave a talk on this at EUC 2016 (https://github.com/duomark/epocxy/blob/master/doc/2016-09-09-EUC-Concurrency-Fount.pdf ). It provides pace regulation via a reservoir of processes that are gated per 1/100th of a second. You would have to arrange to split your data stream to separate processes, but all the pace regulation is there (and implemented as a behaviour, so you could experiment with alternative pacing approaches). jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sun Jan 1 18:39:34 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sun, 01 Jan 2017 17:39:34 +0000 Subject: [erlang-questions] Limit the Network Bandwidth per application In-Reply-To: <1854268120.4982109.1483291044993@mail.yahoo.com> References: <1854268120.4982109.1483291044993@mail.yahoo.com> Message-ID: Hi Vans, I first found "tc" and after bit of thinking what I would like to achieve is much more simpler. Control in/out TCP traffic to a backend server through an Erlang proxy. Thank you for the links. /Frank Le dim. 1 janv. 2017 ? 18:20, Vans S wrote : > This is more difficult then it appears. The standard way is to use cgroups > or window quotas (per user/process). > > > > The other way is to throttle at the interface level, > > > https://www.cyberciti.biz/faq/linux-traffic-shaping-using-tc-to-control-http-traffic/ > . > > > > Yet another way is to use virtualization and set quotas on the > container/vm. > > > > Yet another way is to mark the packets coming from your process so > pf/iptables/tc/other can traffic shape it. > > > > > > Probably the most straightforward way is to simply count every call to > send/recv in Erlang itself and count the amount of bytes sent/recv then do > math. > > > > > > I would probably go for the cgroups or virtualization approach personally. > > > > > > > > > > On Sunday, January 1, 2017 11:30 AM, Frank Muller < > frank.muller.erl@REDACTED> wrote: > > > > > > > > Thank you for the link Jesper. > > > > Actually, any TCP bandwidth limiting (upload, download) code will make it. > > > > I would like to learn how to implement such features in practice. > > > > /Frank > > > > > > Le dim. 1 janv. 2017 ? 14:42, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED> a ?crit : > > > > See https://github.com/jlouis/rlimit but mind you, there is an issue as > well: https://github.com/jlouis/rlimit/issues/2 :) > > > > > >On Sat, Dec 31, 2016 at 9:57 AM Frank Muller > wrote: > > > > > >Hi guys, > > >> > > >>I stumbled across trickle (https://github.com/mariusae/trickle), a > user-land application bandwidth shaper. > > >>A very useful small piece of software as explained here: > > >> > http://www.tecmint.com/manage-and-limit-downloadupload-bandwidth-with-trickle-in-linux/ > > >> > > >> > > >>It relies on libevent (http://libevent.org/), and can be installed vi > apt-get or yum. > > >> > > >>I?m wondering if one can create a trickle equivalent in Erlang (+NIF or > whatever)? > > >> > > >>Feedbacks very appreciated (links to equivalent apps in Erlang, > implementation ideas, etc). > > >> > > >>Thank you and happy new year. > > >>/Frank > > > > > >> > > >>_______________________________________________ > > >> > > >> > > >>erlang-questions mailing list > > >> > > >> > > >>erlang-questions@REDACTED > > >> > > >> > > >>http://erlang.org/mailman/listinfo/erlang-questions > > >> > > >> > > >> > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sun Jan 1 18:57:14 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sun, 01 Jan 2017 17:57:14 +0000 Subject: [erlang-questions] Limit the Network Bandwidth per application In-Reply-To: References: Message-ID: Hi Jay Excellent. "epocxy" will certainly help me understand this load regulation principles. /Frank Le dim. 1 janv. 2017 ? 18:38, Jay Nelson a ?crit : > You might want to take a look at https://github.com/duomark/epocxy for > cxy_fount. I gave a talk on this at EUC 2016 ( > https://github.com/duomark/epocxy/blob/master/doc/2016-09-09-EUC-Concurrency-Fount.pdf). > It provides pace regulation via a reservoir of processes that are gated per > 1/100th of a second. You would have to arrange to split your data stream to > separate processes, but all the pace regulation is there (and implemented > as a behaviour, so you could experiment with alternative pacing approaches). > > jay > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Sun Jan 1 23:29:00 2017 From: jay@REDACTED (Jay Nelson) Date: Sun, 1 Jan 2017 14:29:00 -0800 Subject: [erlang-questions] MacOs Sierra + OTP 19.2 + openssl 1.0.2j from source Message-ID: <662944B8-328C-421A-BDF8-5758D1CD0C46@duomark.com> Well I seem to have finally gotten the upgrade working with a new fresh compile of Erlang/OTP without using Brew. On El Capitan I could use the Swift migration tool headers for openssl, but in Sierra they are completely gone. The following steps were used to install a new version of openssl development headers and erlang: 1) Upgrade to latest Xcode and accept license 2) Add command line tools to Xcode 3) Upgrade to Sierra 4) Download and install openssl 1.0.2j a) ./configure darwin64-x86_64-cc shared (this seemed to be the key step) b) make depend c) make d) sudo make install e) sudo ln -s /usr/local/ssl/bin/openssl /usr/local/bin/openssl f) make sure /usr/local/bin comes before /usr/bin in $PATH g) openssl version => 1.0.2j instead of 0.9.8zh 5) Download and install Erlang 19.2 a) ./configure ?with-ssl=/usr/local/ssl b) make -j6 c) sudo make install d) erl e) 1> l(crypto). f) 2> crypto:strong_rand_bytes(22). OpenSSL 1.1.0 has a very different structure that 1.0.x and probably can?t work with Erlang/OTP until changes are made specifically for it because the code is being split to different directories. The above steps make openSSL 1.0.2j found before the Apple version, not sure what problems that will cause with other software. Also, I don?t know if it is wise to use shared libraries for crypto instead of static. If it didn?t work for your environment, loading the crypto module will cause an error. Any comments about why this is wrong, or where I will run into trouble are highly encouraged! Others may wish to try this approach if I don?t complain about it after a week or so (although I?m not doing much crypto stuff, l(crypto) and strong_rand_bytes seems to work). jay From drormein@REDACTED Mon Jan 2 12:59:57 2017 From: drormein@REDACTED (Dror Mein) Date: Mon, 2 Jan 2017 11:59:57 +0000 (UTC) Subject: [erlang-questions] large cluster problem References: <1250973719.5273987.1483358397513.ref@mail.yahoo.com> Message-ID: <1250973719.5273987.1483358397513@mail.yahoo.com> Hello list, I have a nice distributed erlang application which has around 50 nodes (10 of them are c-nodes), which is running smoothly. I'm on 16B01 on redhat 6.3.all of them share the same mnesia database - but only some of them read/write extensively to it. There is one table 'params' that everyone use to keep a stored set of parameters on one node (disc copies on that node, ram copies on the rest). all is well, except... I want to double the nodes and the lab testing does not get past 65-70 nodes. Here are some questions:* what tools do you use to find the specific problem (it could be the erlang cluster epmd n^2 messages, the shared mnesia cluster, the stupid param table)* any nice quick fixes (running everything as hidden, break the cluster, delete the stupid table)* long-run fixes: switch from mnesia to something else? hire good programmers? Thanks a lot, Dror -------------- next part -------------- An HTML attachment was scrubbed... URL: From walter.weinmann@REDACTED Mon Jan 2 20:29:10 2017 From: walter.weinmann@REDACTED (Walter Weinmann) Date: Mon, 2 Jan 2017 20:29:10 +0100 Subject: [erlang-questions] EEP Mailing List ? Message-ID: I think I sent an EEP to the mailing list eeps@REDACTED on 26.12.2016 - no response so far. Are both EEP mailing list and process still up-to-date? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Mon Jan 2 20:39:28 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Mon, 2 Jan 2017 20:39:28 +0100 Subject: [erlang-questions] EEP Mailing List ? In-Reply-To: References: Message-ID: Ehm... Days between Christmas and New Year are often taken as vacations. People just are not there. First working day was today and even then some people take longer vacation which covers this week too. Friday Jan 6th is red day again. 2017-01-02 20:29 GMT+01:00 Walter Weinmann : > I think I sent an EEP to the mailing list eeps@REDACTED on 26.12.2016 - > no response so far. Are both EEP mailing list and process still up-to-date? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Tue Jan 3 14:48:39 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 3 Jan 2017 14:48:39 +0100 Subject: [erlang-questions] [ANN] Cowboy 2.0.0-pre.4 Message-ID: Hello and happy new year! Cowboy 2.0.0-pre.4 has been released. It is a significant progress over the previous pre-release and contains most of the breaking changes I wanted for Cowboy 2.0. Details on the release can be read at: https://ninenines.eu/articles/cowboy-2.0.0-pre.4/ There will be mostly weekly pre-releases from now until the 2.0 release candidate. They will only be announced on the website and social media. I encourage everyone to try this new version and report any problems (bug, missing features, compatibility issues). A lot of work went into the manual. If you recall the long documentation thread from a few months back, I have put all my suggestions into the Cowboy manual: https://ninenines.eu/docs/en/cowboy/2.0/manual/ and would love to know what you think. I have ignored tickets and PRs for some time to get this release out. I will now start merging and fixing things again. Thanks for your patience. As always, your feedback is most important. To a great Cowboy year, -- Lo?c Hoguin https://ninenines.eu From lenartlad@REDACTED Tue Jan 3 19:09:28 2017 From: lenartlad@REDACTED (Ladislav Lenart) Date: Tue, 3 Jan 2017 19:09:28 +0100 Subject: [erlang-questions] [ANN] Cowboy 2.0.0-pre.4 In-Reply-To: References: Message-ID: <586BE8D8.3000802@volny.cz> Hello. Just my 2 cents after skimming through the new manual. But first a disclaimer: I do not program in Erlang (anymore). I just keep an eye on various projects created using it and love to read the mailing list. I have never used the cowboy web server... With that in mind, the manual is simply excellent! Great work! What I like: + It seems (though I have not verify it) that it is just a static HTML. + The version number is part of the URL. Awesome! I do not dare to count the number of times when I got lost in the Apple documentation or when I have found a link which no longer works. + Detailed explanation of arguments with simple examples. + Oh, the usage examples! What I have not found (though I have not red the entire manual): - A top-level overview where the concepts such as protocols, handlers, behaviors and middlewares are explained (i.e. what is it, when to use it and why and its relations to others). Out of curiosity, why (in cowboy:start_clear) is TransportOpts a list of pairs while ProtocolOpts is a map? Is this to ensure that the user does not swap them unwittingly? Happy New Year and keep up the great work, Ladislav Lenart On 3.1.2017 14:48, Lo?c Hoguin wrote: > Hello and happy new year! > > Cowboy 2.0.0-pre.4 has been released. It is a significant progress over > the previous pre-release and contains most of the breaking changes I > wanted for Cowboy 2.0. > > Details on the release can be read at: > > https://ninenines.eu/articles/cowboy-2.0.0-pre.4/ > > There will be mostly weekly pre-releases from now until the 2.0 release > candidate. They will only be announced on the website and social media. > > I encourage everyone to try this new version and report any problems > (bug, missing features, compatibility issues). > > A lot of work went into the manual. If you recall the long documentation > thread from a few months back, I have put all my suggestions into the > Cowboy manual: https://ninenines.eu/docs/en/cowboy/2.0/manual/ and would > love to know what you think. > > I have ignored tickets and PRs for some time to get this release out. I > will now start merging and fixing things again. Thanks for your patience. > > As always, your feedback is most important. > > To a great Cowboy year, From essen@REDACTED Tue Jan 3 20:00:07 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 3 Jan 2017 20:00:07 +0100 Subject: [erlang-questions] [ANN] Cowboy 2.0.0-pre.4 In-Reply-To: <586BE8D8.3000802@volny.cz> References: <586BE8D8.3000802@volny.cz> Message-ID: <3bb060c7-918c-eab8-55b9-1e075d65cc77@ninenines.eu> Hey, thanks for the feedback! It's not 100% static HTML, the current page's links on the right are populated by a small piece of JS. Everything else is static. On 01/03/2017 07:09 PM, Ladislav Lenart wrote: > What I have not found (though I have not red the entire manual): > - A top-level overview where the concepts such as protocols, handlers, behaviors > and middlewares are explained (i.e. what is it, when to use it and why and its > relations to others). That would be the role of the user guide. There should be a link on the right. I have mixed feelings on the user guide right now, but can't quite point my finger on it just yet. I am also well aware that there isn't much linking between the guide and the manual at the moment. I'll be fixing that when time allows. > Out of curiosity, why (in cowboy:start_clear) is TransportOpts a list of pairs > while ProtocolOpts is a map? Is this to ensure that the user does not swap them > unwittingly? That's because the underlying library expects a list. Part of that is because maps didn't exist at the time, and parts because some transport options can be repeated (the raw options for example). This is the same options as in gen_tcp/ssl, plus a few extra specific to Cowboy. I suppose it would be possible to accept both, but Cowboy would have to almost immediately convert the map to a list. -- Lo?c Hoguin https://ninenines.eu From sdl.web@REDACTED Wed Jan 4 10:25:32 2017 From: sdl.web@REDACTED (Leo Liu) Date: Wed, 04 Jan 2017 17:25:32 +0800 Subject: [erlang-questions] [ANN] Cowboy 2.0.0-pre.4 References: Message-ID: On 2017-01-03 14:48 +0100, Lo?c Hoguin wrote: > Hello and happy new year! Everything is excellent. High-quality software and documentation. Thanks for creating Cowboy. Leo From mailparmalat@REDACTED Thu Jan 5 10:49:22 2017 From: mailparmalat@REDACTED (Steven) Date: Thu, 5 Jan 2017 11:49:22 +0200 Subject: [erlang-questions] gen_sctp - peeloff and child socket closed state Message-ID: Good day, Our environment: Erlang 17.5 64bit - Centos We've implemented sctp client using gen_sctp and associations comes up fine and closes fine. A main process opens a gen_sctp port and connect outgoing associations. Each established association is peeled off and control is handed over to a spawned process. If the spawned process is shutdown orderly and closes its socket, the association is cleaned up properly. The same association (same local IP/port same remote IP port) can be re-established via the main process (connect, peeloff and then control is passed to spawned process) Under certain conditions, the spawned process would terminate and close its socket, but the association is not cleaned up entirely. Instead control of the association is handed back to the main process (can see events for the association being received by the main process). Attempt would be made to connect but comes back with ealready (operation in process) or eisconn (socket is already connected). On OS, the sctp association be seen in CLOSE state. It is only when the main process closes the original port is when the 'child socket' is released. We've managed to simulating this by pulling the lan cable out on the client side and the association lands up in CLOSE state. Once the cable is plugged in again, the association events for that association ID are received by the main process. These are the following questions I have: 1. When closing the peeloff socket, does that not cleanup the underlying socket and its association? 2. Is this intended behaviour? Any response would be appreciated. Thanks Regards, Steven -------------- next part -------------- An HTML attachment was scrubbed... URL: From silviu.cpp@REDACTED Thu Jan 5 11:39:39 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Thu, 5 Jan 2017 12:39:39 +0200 Subject: [erlang-questions] Downgrade ssl connection Message-ID: Hello, I'm trying to create an example where I send some data over ssl then I downgrade the connection and send data over tcp . Unfortunately I'm doing something wrong. -export([test/0]). get_certificate() -> "test/server.pem". -define(SSL, ssl). test() -> ?SSL:start(), Port = 12000, Opt = [ binary, {exit_on_close, false}, {active, false}, {ciphers, ["AES128-GCM-SHA256"]}, {verify, verify_none} ], {ok, LSocket} = ?SSL:listen(Port, [{certfile, get_certificate()} | Opt]), ClientProc = fun() -> {ok, CSocket} = ?SSL:connect("127.0.0.1", Port, Opt), ok = ?SSL:send(CSocket, <<"PING">>), {ok, <<"PONG">> } = ?SSL:recv(CSocket, 0), {ok, TcpSocket} = ?SSL:close(CSocket, {self(), 3000}), {ok, <<"PLAIN_DATA">>} = gen_tcp:recv(TcpSocket, 0), ok = gen_tcp:send(TcpSocket, <<"PLAIN_RESPONSE">>), ok = gen_tcp:close(CSocket) end, spawn(ClientProc), {ok, Socket} = ?SSL:transport_accept(LSocket, 5000), ok = ?SSL:ssl_accept(Socket), {ok, <<"PING">> } = ?SSL:recv(Socket, 0), ok = ?SSL:send(Socket, <<"PONG">>), {ok, TcpSocket} = ?SSL:close(Socket, {self(), 3000}), ok = gen_tcp:send(TcpSocket, <<"PLAIN_DATA">>), {ok, <<"PLAIN_RESPONSE">>} = gen_tcp:recv(TcpSocket, 0), ok = gen_tcp:close(Socket), ok = ?SSL:close(LSocket), true. Any idea why it's not properly working ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Thu Jan 5 15:20:48 2017 From: roger@REDACTED (Roger Lipscombe) Date: Thu, 5 Jan 2017 14:20:48 +0000 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) Message-ID: We're using ECDHE and DHE ciphers for our SSL connections. This provides perfect forward secrecy, which is good, but it makes it impossible to decipher packet captures in wireshark, which is expected, and also good, almost all of the time. Sometimes, however, we *do* need to decipher the traffic. Note that we own both the client (which is embedded) and the server (which uses Erlang -- otherwise I wouldn't be asking here -- and ranch). We *could* offer a different cipher suite on the server, which would disable PFS, but would do it for all connections. I'd prefer something a bit more fine-grained. You can feed a key log to Wireshark, as documented at https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format, and it'll correctly decipher the traffic for that connection. I'd like to find a way to generate a key log file. This requires either (client_random, master_secret) or (encrypted_premaster_secret, premaster_secret). Note that I'm looking at the OTP 17.5 source, because that's what we're using. It would seem that premaster_secret is not stored past the initial negotiation, but the client_random and master_secret values are in the #security_parameters record in the #connection_state record in the #connection_states record, which is in the #state record of the SSL connection pid. But I can't see any (clean) way to retrieve these values, in order to generate a key log suitable for Wireshark. Is there any clean way to do this in OTP 17.5, or is there a supported way to do this in OTP 18.x or 19.x? Regards, Roger. From roger@REDACTED Thu Jan 5 15:24:07 2017 From: roger@REDACTED (Roger Lipscombe) Date: Thu, 5 Jan 2017 14:24:07 +0000 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: Further searching shows that Ingela rejected something related in http://erlang.org/pipermail/erlang-patches/2012-February/002681.html, but I'm not sure whether that was about exposing the key material or the way that that patch should have been implemented as ssl:prf (which it eventually was). On 5 January 2017 at 14:20, Roger Lipscombe wrote: > We're using ECDHE and DHE ciphers for our SSL connections. This > provides perfect forward secrecy, which is good, but it makes it > impossible to decipher packet captures in wireshark, which is > expected, and also good, almost all of the time. > > Sometimes, however, we *do* need to decipher the traffic. > > Note that we own both the client (which is embedded) and the server > (which uses Erlang -- otherwise I wouldn't be asking here -- and > ranch). We *could* offer a different cipher suite on the server, which > would disable PFS, but would do it for all connections. I'd prefer > something a bit more fine-grained. > > You can feed a key log to Wireshark, as documented at > https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format, > and it'll correctly decipher the traffic for that connection. > > I'd like to find a way to generate a key log file. This requires > either (client_random, master_secret) or (encrypted_premaster_secret, > premaster_secret). > > Note that I'm looking at the OTP 17.5 source, because that's what we're using. > > It would seem that premaster_secret is not stored past the initial > negotiation, but the client_random and master_secret values are in the > #security_parameters record in the #connection_state record in the > #connection_states record, which is in the #state record of the SSL > connection pid. > > But I can't see any (clean) way to retrieve these values, in order to > generate a key log suitable for Wireshark. > > Is there any clean way to do this in OTP 17.5, or is there a supported > way to do this in OTP 18.x or 19.x? > > Regards, > Roger. From walter.weinmann@REDACTED Thu Jan 5 15:24:39 2017 From: walter.weinmann@REDACTED (Walter Weinmann) Date: Thu, 5 Jan 2017 15:24:39 +0100 Subject: [erlang-questions] Mnesia Version 4.14 - Documentation Issue Message-ID: In this version, an external backend plugin api was announced, but it seems that even in OTP 19.2 still the documentation is missing - or am I wrong here? -------------- next part -------------- An HTML attachment was scrubbed... URL: From technion@REDACTED Fri Jan 6 00:13:35 2017 From: technion@REDACTED (Technion) Date: Thu, 5 Jan 2017 23:13:35 +0000 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: Hi, Is it a solution to for you to deal with it on the client side? https://jimshaver.net/2015/02/11/decrypting-tls-browser-traffic-with-wireshark-the-easy-way/ Chrome lets you write keys out. ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Roger Lipscombe Sent: Friday, 6 January 2017 1:20 AM To: erlang-questions@REDACTED Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) We're using ECDHE and DHE ciphers for our SSL connections. This provides perfect forward secrecy, which is good, but it makes it impossible to decipher packet captures in wireshark, which is expected, and also good, almost all of the time. Sometimes, however, we *do* need to decipher the traffic. Note that we own both the client (which is embedded) and the server (which uses Erlang -- otherwise I wouldn't be asking here -- and ranch). We *could* offer a different cipher suite on the server, which would disable PFS, but would do it for all connections. I'd prefer something a bit more fine-grained. You can feed a key log to Wireshark, as documented at https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format, [https://developer.cdn.mozilla.net/static/img/opengraph-logo.dc4e08e2f6af.png] NSS Key Log Format - Mozilla | MDN developer.mozilla.org Key logs can be written by NSS so that external programs can decrypt TLS connections. Wireshark 1.6.0 and above can use these log files to decrypt packets. and it'll correctly decipher the traffic for that connection. I'd like to find a way to generate a key log file. This requires either (client_random, master_secret) or (encrypted_premaster_secret, premaster_secret). Note that I'm looking at the OTP 17.5 source, because that's what we're using. It would seem that premaster_secret is not stored past the initial negotiation, but the client_random and master_secret values are in the #security_parameters record in the #connection_state record in the #connection_states record, which is in the #state record of the SSL connection pid. But I can't see any (clean) way to retrieve these values, in order to generate a key log suitable for Wireshark. Is there any clean way to do this in OTP 17.5, or is there a supported way to do this in OTP 18.x or 19.x? Regards, Roger. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions erlang-questions Info Page erlang.org Mailing list for general discussions about Erlang/OTP, the language, implementation, usage, beginners questions, etc... To see the collection of prior postings to the ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Jan 6 04:05:42 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 6 Jan 2017 16:05:42 +1300 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <1482615262.7413280@apps.rackspace.com> References: <1482615262.7413280@apps.rackspace.com> Message-ID: On 25/12/16 10:34 AM, lloyd@REDACTED wrote: > Suppose I have the following string: > > "

Hello!

\n

How are you?

\n

Some text\n and more text.

" > > I would like to break it into a list: > > ["

Hello!

", "

How are you?

", "

Some text\n and more text.

"] > > string:token(MyString, "$\n") doesn't work because it would break the paragraph. So you don't have "a string", you have an XML fragment that happens to be stored as a string. My problem in reading this is that I don't have the faintest idea what you want to happen IN GENERAL. - What is to happen if there is a newline character in an attribute? "Two gorillas\One cop - What is to happen if there is a newline between tokens inside a tag? "Anchor Text" - What is to happen if there is a newline inside an element other than a

element? "

Two gorillas, one cop
\nSpoof movie of the year

" - What is to happen if there AREN'T newlines? "

Hello!

How are you?

Some text\n etc.

" White space between block level elements isn't significant, which means that you can't in general expect it to be there or to be preserved by other tools. - What is to happen if the newlines between elements are doubled? "

Hello!

\n\n

How are you?

\n\n

Some text\n and more text.

" - What is to happen if there is a newline whitespace sequence at the end? "

Hello!

\n

How are you?

\n

Some text\n and more text.

\n " If I needed to do this, I'd look for an XML library in which I could do Fragment = xml:parse_fragment(String), [xml:unparse_element(Element) || Element <- Fragment] (At least, that's my GUESS about what you want to achieve.) To me, this seems like a textbook example of why Strings Are Wrong and regular expressions make it incredibly easy to do the wrong thing. From frank.muller.erl@REDACTED Sat Jan 7 12:04:09 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 07 Jan 2017 11:04:09 +0000 Subject: [erlang-questions] inets {read_packets,N} option Message-ID: Hi guys, Can someone explain me the semantic behind {read_packets, Integer} for UDP: http://erlang.org/doc/man/inet.html I understand the associated doc, but what i can?t get is how this option affects me if i set it for example with: 1. {read_packets, 20} + {active, once} 2. {read_packets, 20} + {active, 100} How many packets my process will receive in each case? And are these packets send as multiple messages one packet at a time, or as one message representing a list of N packets? Thank you /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Sat Jan 7 22:05:31 2017 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Sat, 7 Jan 2017 23:05:31 +0200 Subject: [erlang-questions] xmerl and the xml namespace Message-ID: Hello erlangers, I tried to parse XML document containing namespaced elements using xmerl. The document contain xml:lang attributes and no declaration for the xml prefix. The minimal example reproducing the error is bellow. 1> catch xmerl_scan:string("
", [{namespace_conformant, true}]). {'EXIT',{fatal,{{namespace_prefix_not_declared,"xml"}, {file,file_name_unknown}, {line,1}, {col,23}}}} The same XML document is parsed successfully with {namespace_conformant, false}. The XML specification (https://www.w3.org/TR/xml-names/#xmlReserved) says, that the xml prefix can be used without declaration and must refer to "http://www.w3.org/XML/1998/namespace". Is this a bug in xmerl, or I misunderstand something? Karolis From alex0player@REDACTED Sat Jan 7 22:17:10 2017 From: alex0player@REDACTED (Alex S.) Date: Sun, 8 Jan 2017 00:17:10 +0300 Subject: [erlang-questions] inets {read_packets,N} option In-Reply-To: References: Message-ID: {read_packets, X} refers to number of UDP packets that are read whenever the socket is notified "ready". So if 100 UDP packets arrive at the same time, only 5 will be read until a new one arrives. There isn't really a way to know how many dgrams are waiting on a socket, so that's an anti-flood option. {active, X} refers to number of UDP packets that are immediately sent to Erlang process instead of being read into an internal buffer. 2017-01-07 14:04 GMT+03:00 Frank Muller : > Hi guys, > > Can someone explain me the semantic behind {read_packets, Integer} for UDP: > http://erlang.org/doc/man/inet.html > > I understand the associated doc, but what i can?t get is how this option > affects me if i set it for example with: > 1. {read_packets, 20} + {active, once} > 2. {read_packets, 20} + {active, 100} > > How many packets my process will receive in each case? > > And are these packets send as multiple messages one packet at a time, or > as one message representing a list of N packets? > > Thank you > /Frank > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Jan 7 22:39:06 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 07 Jan 2017 21:39:06 +0000 Subject: [erlang-questions] inets {read_packets,N} option In-Reply-To: References: Message-ID: Hi Alex Still didn't get it. In your example 100 UDP packets arrive so the socket's ready for reading. Then, N=5 are read out of 100. Why you said 5 will be read until new one arrives? There's still 95 ready for reading right away after delivering the first 5. I'm right? Thank you. /Frank Le sam. 7 janv. 2017 ? 22:17, Alex S. a ?crit : > {read_packets, X} refers to number of UDP packets that are read whenever > the socket is notified "ready". So if 100 UDP packets arrive at the same > time, only 5 will be read until a new one arrives. There isn't really a way > to know how many dgrams are waiting on a socket, so that's an anti-flood > option. > {active, X} refers to number of UDP packets that are immediately sent to > Erlang process instead of being read into an internal buffer. > > 2017-01-07 14:04 GMT+03:00 Frank Muller : > > Hi guys, > > Can someone explain me the semantic behind {read_packets, Integer} for UDP: > http://erlang.org/doc/man/inet.html > > I understand the associated doc, but what i can?t get is how this option > affects me if i set it for example with: > 1. {read_packets, 20} + {active, once} > 2. {read_packets, 20} + {active, 100} > > How many packets my process will receive in each case? > > And are these packets send as multiple messages one packet at a time, or > as one message representing a list of N packets? > > Thank you > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sun Jan 8 16:29:45 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sun, 08 Jan 2017 15:29:45 +0000 Subject: [erlang-questions] [ANN] geas 2.0.10 In-Reply-To: <65147801-91b6-f4b4-097c-1e3125df3d37@wanadoo.fr> References: <65147801-91b6-f4b4-097c-1e3125df3d37@wanadoo.fr> Message-ID: Hi Eric, I think I was able to understand the bug (in geas.erl): get_src_from_beam(File) -> UpperDir = filename:dirname(filename:dirname(File)), Basename = filename:rootname(filename:basename(File)), case string:substr(Basename, string:len(Basename) -3 ) of %% SrcDir = filename:join(UpperDir, "templates"), Dtl = string:substr(Basename, 1, string:len(Basename) -4 ), filename:join([SrcDir, Dtl ++ ".dtl"]); _ -> SrcDir = filename:join(UpperDir, "src"), filename:join([SrcDir, Basename, ".erl"]) end. You're substracting -3 from the module's basename length. Because my module is called "dir.erl" (only 3 chars), this returns 0. Then, call to string:substr/2 crashes because second argument must be > 0. Hope this helps. /Frank Le sam. 7 janv. 2017 ? 21:55, PAILLEAU Eric a ?crit : > Hi Franck, > > > > did you finally solved your problem ? > > because no issue opened... > > > > My main concern is to propose quality projects, > > so an issue always interest me ! > > > > you are welcome ! > > Regards > > > > > > > > Le 31/12/2016 ? 15:51, Frank Muller a ?crit : > > > Yes I did compile my project first. > > > All the .beam are in ./ebin and deps/*/ebin. > > > > > > Ok for the ticket on GitHub. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sun Jan 8 16:31:49 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 08 Jan 2017 15:31:49 +0000 Subject: [erlang-questions] inets {read_packets,N} option In-Reply-To: References: Message-ID: Packet delivery happens roughly as follows: 1. The network card gets the packet. 2. The network card stores the packet in a kernel buffer. 3. The Erlang VM reads packets out of the kernel buffer and into its own memory space. This opens up the kernel buffer for more UDP packets to arrive. 4. The Erlang VM sends a message to a mailbox of a process with the data. Rules: * If the kernel buffer is full, new arrivals are dropped on the floor. * The kernel buffer is set separately, often with an operating system default. * The VM won't flood a process unless it has {active, true} set. {read_packets, K} controls what happens in 3. in the above. If set at 5, once the kernel tells the VM there are data available to read, the VM will read out 5 packets. It won't read more packets until a new packet arrives at the socket in which case it will read up to 5 packets more. Note that if the arrival rate is larger than the consumption rate then packets will eventually gets dropped on the floor. With UDP, your responsibility is to handle this situation. {active, N} controls how many packets can pre-delivered into the mailbox with no signs of progress from the process itself. It corresponds to a limiter at 4. in the above. You have situations when N < K, N > K and N == K to think about here. In those situations, you may end up with lost packets if the system goes over the capacity it is able to systematically handle. The kernel tracks stats for how many packets it has thrown away. I don't know if the Erlang VM does somewhere (it should, provided that it throws data on the floor). On Sat, Jan 7, 2017 at 10:39 PM Frank Muller wrote: > Hi Alex > > Still didn't get it. In your example 100 UDP packets arrive so the > socket's ready for reading. Then, N=5 are read out of 100. > > Why you said 5 will be read until new one arrives? > There's still 95 ready for reading right away after delivering the first > 5. I'm right? > > Thank you. > /Frank > > Le sam. 7 janv. 2017 ? 22:17, Alex S. a ?crit : > > {read_packets, X} refers to number of UDP packets that are read whenever > the socket is notified "ready". So if 100 UDP packets arrive at the same > time, only 5 will be read until a new one arrives. There isn't really a way > to know how many dgrams are waiting on a socket, so that's an anti-flood > option. > {active, X} refers to number of UDP packets that are immediately sent to > Erlang process instead of being read into an internal buffer. > > 2017-01-07 14:04 GMT+03:00 Frank Muller : > > Hi guys, > > Can someone explain me the semantic behind {read_packets, Integer} for UDP: > http://erlang.org/doc/man/inet.html > > I understand the associated doc, but what i can?t get is how this option > affects me if i set it for example with: > 1. {read_packets, 20} + {active, once} > 2. {read_packets, 20} + {active, 100} > > How many packets my process will receive in each case? > > And are these packets send as multiple messages one packet at a time, or > as one message representing a list of N packets? > > Thank you > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ameretat.reith@REDACTED Sun Jan 8 19:43:26 2017 From: ameretat.reith@REDACTED (Ameretat Reith) Date: Sun, 8 Jan 2017 22:13:26 +0330 Subject: [erlang-questions] inets {read_packets,N} option In-Reply-To: References: Message-ID: <20170108221326.6c1e00f9@gmail.com> > > 2017-01-07 14:04 GMT+03:00 Frank Muller > > : > > > > I understand the associated doc, but what i can?t get is how this > > option affects me if i set it for example with: > > 1. {read_packets, 20} + {active, once} > > 2. {read_packets, 20} + {active, 100} 1: Once there is data in socket, you'll get it and then your socket will not remain active anymore. you'll get another packet when you set socket as active again and there is a more data. 2: Once there is data in socket, up to twenty datagrams will be read and will be passed to target process. If there is more data, It'll be processed on next polling iteration over and over until socket is not active anymore. It's like this: while true: if socket_is_active(): // this is blocking polling function, a // select/poll/epoll_wait loop if there_is_data_for_reading_in_socket(): N = 0 while N < read_packets: packet = non_blocking_read() if socket_had_data(packet): N-- pass_to_process(packet) update_active_status() else: break Basically there is a polling loop to select/poll/epoll on set of file descriptors. There is also a number for packets that will be read if polling loop notifies that socket has data, it is read_packets. After certain number of packets is read, socket will be passive; it means looping function won't poll for that socket anymore, that number is X in {active, X}. > > How many packets my process will receive in each case? > > > > And are these packets send as multiple messages one packet at a > > time, or as one message representing a list of N packets? One message for each datagram (packet). > > Still didn't get it. In your example 100 UDP packets arrive so the > > socket's ready for reading. Then, N=5 are read out of 100. > > > > Why you said 5 will be read until new one arrives? > > There's still 95 ready for reading right away after delivering the > > first 5. I'm right? Assuming socket's read_packet is set to 5, 5 packets will be read and on next polling iterations, buffered packets will be processed. There is no need for more packets since there already are in kernel buffer. > On Sun, 08 Jan 2017 15:31:49 +0000 > Jesper Louis Andersen wrote: > Packet delivery happens roughly as follows: > {read_packets, K} controls what happens in 3. in the above. If set at > 5, once the kernel tells the VM there are data available to read, the > VM will read out 5 packets. It won't read more packets until a new > packet arrives at the socket in which case it will read up to 5 > packets more. Actually It will read more packets if socket is still active and there is data. Documentation is a little ambiguous too: > {read_packets, Integer}(UDP sockets) > Sets the max number of UDP packets to read without intervention from > the socket when data is available. When this many packets have been > read and delivered to the destination process, new packets are not > read until a new notification of available data has arrived. Here, *notification of available data* does not mean expectance for *new* data. I think read_packets is more useful to overcome blocking call overhead of polling functions rather than being an flood protection solution. If we are experiencing burst of packets, reads will be delayed a little but will still affect and now with more polling syscalls too. From frank.muller.erl@REDACTED Sun Jan 8 20:13:34 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sun, 08 Jan 2017 19:13:34 +0000 Subject: [erlang-questions] inets {read_packets,N} option In-Reply-To: <20170108221326.6c1e00f9@gmail.com> References: <20170108221326.6c1e00f9@gmail.com> Message-ID: Thank you guys. Now I get it. All this (or a summary) should go to the doc because it isn't so clear IMHO. /Frank Le dim. 8 janv. 2017 ? 19:44, Ameretat Reith a ?crit : > > > 2017-01-07 14:04 GMT+03:00 Frank Muller > > > > : > > > > > > > > > > I understand the associated doc, but what i can?t get is how this > > > > option affects me if i set it for example with: > > > > 1. {read_packets, 20} + {active, once} > > > > 2. {read_packets, 20} + {active, 100} > > > > 1: > > Once there is data in socket, you'll get it and then your socket will > > not remain active anymore. you'll get another packet when you set socket > > as active again and there is a more data. > > > > 2: > > Once there is data in socket, up to twenty datagrams will be read and > > will be passed to target process. If there is more data, It'll be > > processed on next polling iteration over and over until socket is not > > active anymore. > > > > > > It's like this: > > > > while true: > > if socket_is_active(): > > > > // this is blocking polling function, a > > // select/poll/epoll_wait loop > > if there_is_data_for_reading_in_socket(): > > > > N = 0 > > while N < read_packets: > > > > packet = non_blocking_read() > > > > if socket_had_data(packet): > > N-- > > > > pass_to_process(packet) > > > > update_active_status() > > else: > > break > > > > > > Basically there is a polling loop to select/poll/epoll on set of file > > descriptors. There is also a number for packets that will be read if > > polling loop notifies that socket has data, it is read_packets. > > After certain number of packets is read, socket will be passive; it > > means looping function won't poll for that socket anymore, that number > > is X in {active, X}. > > > > > > How many packets my process will receive in each case? > > > > > > > > And are these packets send as multiple messages one packet at a > > > > time, or as one message representing a list of N packets? > > > > One message for each datagram (packet). > > > > > > Still didn't get it. In your example 100 UDP packets arrive so the > > > > socket's ready for reading. Then, N=5 are read out of 100. > > > > > > > > Why you said 5 will be read until new one arrives? > > > > There's still 95 ready for reading right away after delivering the > > > > first 5. I'm right? > > > > Assuming socket's read_packet is set to 5, 5 packets will be read and > > on next polling iterations, buffered packets will be processed. There > > is no need for more packets since there already are in kernel buffer. > > > > > On Sun, 08 Jan 2017 15:31:49 +0000 > > > Jesper Louis Andersen wrote: > > > > > Packet delivery happens roughly as follows: > > > {read_packets, K} controls what happens in 3. in the above. If set at > > > 5, once the kernel tells the VM there are data available to read, the > > > VM will read out 5 packets. It won't read more packets until a new > > > packet arrives at the socket in which case it will read up to 5 > > > packets more. > > > > Actually It will read more packets if socket is still active and there > > is data. Documentation is a little ambiguous too: > > > > > {read_packets, Integer}(UDP sockets) > > > Sets the max number of UDP packets to read without intervention from > > > the socket when data is available. When this many packets have been > > > read and delivered to the destination process, new packets are not > > > read until a new notification of available data has arrived. > > > > Here, *notification of available data* does not mean expectance for > > *new* data. > > > > I think read_packets is more useful to overcome blocking call overhead > > of polling functions rather than being an flood protection solution. If > > we are experiencing burst of packets, reads will be delayed a little but > > will still affect and now with more polling syscalls too. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Sun Jan 8 20:31:26 2017 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Sun, 8 Jan 2017 11:31:26 -0800 Subject: [erlang-questions] inets {read_packets,N} option In-Reply-To: References: <20170108221326.6c1e00f9@gmail.com> Message-ID: <112df820-2c76-d6af-ff30-fed03069fbd2@gmail.com> On 01/08/2017 11:13 AM, Frank Muller wrote: > All this (or a summary) should go to the doc because it isn't so clear > IMHO. If you get the itch to write up a correction, good pull requests are accepted at . The file containing the documentation that you're concerned about is at $ERL_TOP/lib/kernel/doc/src/inet.xml (But read and probably first!) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From raimo+erlang-questions@REDACTED Tue Jan 10 10:40:27 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 10 Jan 2017 10:40:27 +0100 Subject: [erlang-questions] EEP Mailing List ? In-Reply-To: References: Message-ID: <20170110094027.GA15677@erix.ericsson.se> On Mon, Jan 02, 2017 at 08:39:28PM +0100, Dmytro Lytovchenko wrote: > Ehm... Days between Christmas and New Year are often taken as vacations. > People just are not there. First working day was today and even then some > people take longer vacation which covers this week too. Friday Jan 6th is > red day again. Correct. I came back from vacation yesterday and is catching up. The EEP will be added today. > > 2017-01-02 20:29 GMT+01:00 Walter Weinmann : > > > I think I sent an EEP to the mailing list eeps@REDACTED on 26.12.2016 - > > no response so far. Are both EEP mailing list and process still up-to-date? > > > > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From walter.weinmann@REDACTED Tue Jan 10 10:43:56 2017 From: walter.weinmann@REDACTED (Walter Weinmann) Date: Tue, 10 Jan 2017 10:43:56 +0100 Subject: [erlang-questions] EEP Mailing List ? In-Reply-To: <20170110094027.GA15677@erix.ericsson.se> References: <20170110094027.GA15677@erix.ericsson.se> Message-ID: Thanks. On 10 January 2017 at 10:40, Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > On Mon, Jan 02, 2017 at 08:39:28PM +0100, Dmytro Lytovchenko wrote: > > Ehm... Days between Christmas and New Year are often taken as vacations. > > People just are not there. First working day was today and even then some > > people take longer vacation which covers this week too. Friday Jan 6th is > > red day again. > > Correct. > > I came back from vacation yesterday and is catching up. The EEP will be > added today. > > > > > > 2017-01-02 20:29 GMT+01:00 Walter Weinmann : > > > > > I think I sent an EEP to the mailing list eeps@REDACTED on > 26.12.2016 - > > > no response so far. Are both EEP mailing list and process still > up-to-date? > > > > > > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dpheasant@REDACTED Tue Jan 10 17:47:24 2017 From: dpheasant@REDACTED (Pheasant, David T.) Date: Tue, 10 Jan 2017 16:47:24 +0000 Subject: [erlang-questions] Erlang crashes when using -name Message-ID: Hello everyone, I'm having a very weird issue with erlang that I'm hoping you all may be able to shed some light on. Long story short, we're using a 3rd party product that in turn uses RabbitMQ, which as we all know, uses Erlang. RabbitMQ will not start and I've tracked the issue down to the point where Erlang will start when a short node name is used (-sname rabbit@%computername%), but will not start with a long node name (-name rabbit@%computername%.domain.com). I've tested forward and reverse lookups of the FQDN via nslookup and the correct address is returned in all cases. I've also put the FQDN and hostname in the hosts file, which did not change the behavior. When using the long name, the erl.exe process dies immediately and does not print any error messages to the console. When it does die, Windows runs WERFault.exe, shows the familiar "erl.exe has stopped working" message, and creates the following entry in the event log: Faulting application name: erl.exe, version: 0.0.0.0, time stamp: 0x52e15856 Faulting module name: MSVCR100.dll, version: 10.0.40219.1, time stamp: 0x4d5f034a Exception code: 0xc0000005 Fault offset: 0x000000000003c120 Faulting process id: 0x1344 Faulting application start time: 0x01d26b5c79195ddd Faulting application path: C:\Program Files\ArcGIS\Server\framework\runtime\rabbitmq\erlang\bin\erl.exe Faulting module path: C:\Windows\SYSTEM32\MSVCR100.dll Report Id: b6d11c0f-d74f-11e6-80e1-00505607156a Faulting package full name: Other pertinent information: * Operating system is Windows Server 2012 * Erlang version is 5.10.4 * Host is a virtual server (vmware) * Host was migrated from on-prem to IaaS service, during which MAC address changed * IPv6 is disabled on the network adapter * Separate/isolated install of erlang 5.10.4 exhibits same behavior, so this 'feels' like a hardware/OS issue and not software configuration * When running EPMD in debug mode and using a short name, I can see erl.exe register. When using a long name, it appears that erl.exe does not attempt to register (or fails). So, my questions: * Has anyone seen this before? * If not, what differs in the startup process of erl.exe when a short name is used vs. a long name? * Could a change in MAC address have any effect on this? * What (if any) checks does erlang do when started with a long name vs short name (DNS wise)? Thanks in advance for all of your help, -David -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhag.chandra.erl@REDACTED Tue Jan 10 19:58:56 2017 From: bhag.chandra.erl@REDACTED (Bhag Chandra) Date: Wed, 11 Jan 2017 00:28:56 +0530 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead Message-ID: Hello, I have been coding in Erlang for 2 years. A wonderful language but not very big community, so I cant discuss my questions with programmers around me (Java, Python guys). I found out about this list today. I have some fundamental doubts about the Erlang. It would be great if someone can help me clarify them. 1) "Strings in Erlang are internally treated as a list of integers of each character's ASCII values, this representation of string makes operations faster. For example, string concatenation is constant time operation in Erlang." Can someone explain why? 2) "It makes sense to use Erlang only where system's availability is very high". Is it not a very general requirement of most of the systems? Whatsapp to Google to FB to Amazon to Paypal to Barclays etc they all are high availability systems, so we can use Erlang in all of them? 3) "Every message which is sent to a process, goes to the mailbox of that process. When process is free, it consumes that message from mailbox". So how exactly does process ask from the mailbox for that message? Is there a mechanism in a process' memory which keeps polling its mailbox. I basically want to understand how message is sent from mailbox to my code in process. 4) We say that a message is passed from process A to process B by simply using a bang (!) character, but what happens behind the scenes to pass this message? Do both processes establish a tcp connection first and then pass message or what? 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) Mr. Armstrong is talking about the difference between the context switching overhead between OS threads and Erlang processes. He says, thread context switching is of order 700 words but Erlang process context switching is ... ? I cant understand what he said, if someone could tell. P.S. Please excuse for any grammatical errors, English is not my first language. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo@REDACTED Tue Jan 10 21:03:56 2017 From: hugo@REDACTED (Hugo Mills) Date: Tue, 10 Jan 2017 20:03:56 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: <20170110200356.GM19585@carfax.org.uk> On Wed, Jan 11, 2017 at 12:28:56AM +0530, Bhag Chandra wrote: > I have been coding in Erlang for 2 years. A wonderful language but not > very big community, so I cant discuss my questions with programmers around > me (Java, Python guys). I found out about this list today. > > I have some fundamental doubts about the Erlang. It would be great if > someone can help me clarify them. I can answer some of them. > 1) "Strings in Erlang are internally treated as a list of integers of each > character's ASCII values, this representation of string makes operations > faster. For example, string concatenation is constant time operation in > Erlang." Can someone explain why? That's not right. String concatenation is O(n), not O(1). When constructing the concatenation A ++ B, the list A must be traversed in order to place each character at the head of the result list. > 2) "It makes sense to use Erlang only where system's availability is very > high". Is it not a very general requirement of most of the systems? > Whatsapp to Google to FB to Amazon to Paypal to Barclays etc they all are > high availability systems, so we can use Erlang in all of them? > > 3) "Every message which is sent to a process, goes to the mailbox of that > process. When process is free, it consumes that message from mailbox". So > how exactly does process ask from the mailbox for that message? Is there a > mechanism in a process' memory which keeps polling its mailbox. I basically > want to understand how message is sent from mailbox to my code in process. This is the "receive" expression. It runs a set of matches against the mailbox and returns the first matching message. > 4) We say that a message is passed from process A to process B by simply > using a bang (!) character, but what happens behind the scenes to pass this > message? Do both processes establish a tcp connection first and then pass > message or what? Erlang processes aren't the same thing as OS processes. Typically, you'll have one VM (a node), which will run anywhere between dozens and millions of erlang processes. This all takes place under one OS process -- the VM. Communications between erlang processes are handled internally to the VM. If you're doing distributed erlang, you'll have several nodes (possibly, but not always, on the different machines). These nodes connect to each other with permanent TCP connections, over which they talk the erlang distribution protocol. Hugo. -- Hugo Mills | Great films about cricket: Batsman Begins hugo@REDACTED carfax.org.uk | starring Christian Bail http://carfax.org.uk/ | PGP: E2AB1DE4 | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From kennethlakin@REDACTED Tue Jan 10 21:48:31 2017 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Tue, 10 Jan 2017 12:48:31 -0800 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <20170110200356.GM19585@carfax.org.uk> References: <20170110200356.GM19585@carfax.org.uk> Message-ID: <231f04dc-2552-b0bf-bffb-6e44190e183e@gmail.com> On 01/10/2017 12:03 PM, Hugo Mills wrote: > This is the "receive" expression. It runs a set of matches against > the mailbox and returns the first matching message. I think of receive as being similar to select(2). You have a data source you're interested in pulling data from (your mailbox), and you can choose to either wait up to a certain amount of time for new data (receive ... after T ...) or you can block your process until new data appears. Of course -as you mention- receive lets you pattern match on messages, so you aren't forced to process messages that are not of interest. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From k.petrauskas@REDACTED Tue Jan 10 21:53:35 2017 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Tue, 10 Jan 2017 22:53:35 +0200 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <20170110200356.GM19585@carfax.org.uk> References: <20170110200356.GM19585@carfax.org.uk> Message-ID: Hi, On Tue, Jan 10, 2017 at 10:03 PM, Hugo Mills wrote: > On Wed, Jan 11, 2017 at 12:28:56AM +0530, Bhag Chandra wrote: >> I have been coding in Erlang for 2 years. A wonderful language but not >> very big community, so I cant discuss my questions with programmers around >> me (Java, Python guys). I found out about this list today. >> >> I have some fundamental doubts about the Erlang. It would be great if >> someone can help me clarify them. > > I can answer some of them. > >> 1) "Strings in Erlang are internally treated as a list of integers of each >> character's ASCII values, this representation of string makes operations >> faster. For example, string concatenation is constant time operation in >> Erlang." Can someone explain why? > > That's not right. String concatenation is O(n), not O(1). When > constructing the concatenation A ++ B, the list A must be traversed in > order to place each character at the head of the result list. You can have constant time concatenation if you can consider your strings as iolist() (which you can on most IO functions). Example: A = "hello", B = "world", C = [A , " ", C]. io:format("~s", [C]). % This will print "hello world" > >> 2) "It makes sense to use Erlang only where system's availability is very >> high". Is it not a very general requirement of most of the systems? >> Whatsapp to Google to FB to Amazon to Paypal to Barclays etc they all are >> high availability systems, so we can use Erlang in all of them? For me its usually much easier to develop a system in Erlang, that other language, even if it has no high availability requirements. Despite my 12 years experience in Java. >> 3) "Every message which is sent to a process, goes to the mailbox of that >> process. When process is free, it consumes that message from mailbox". So >> how exactly does process ask from the mailbox for that message? Is there a >> mechanism in a process' memory which keeps polling its mailbox. I basically >> want to understand how message is sent from mailbox to my code in process. > > This is the "receive" expression. It runs a set of matches against > the mailbox and returns the first matching message. > >> 4) We say that a message is passed from process A to process B by simply >> using a bang (!) character, but what happens behind the scenes to pass this >> message? Do both processes establish a tcp connection first and then pass >> message or what? > > Erlang processes aren't the same thing as OS processes. Typically, > you'll have one VM (a node), which will run anywhere between dozens > and millions of erlang processes. This all takes place under one OS > process -- the VM. Communications between erlang processes are handled > internally to the VM. > > If you're doing distributed erlang, you'll have several nodes > (possibly, but not always, on the different machines). These nodes > connect to each other with permanent TCP connections, over which they > talk the erlang distribution protocol. > > Hugo. > > -- > Hugo Mills | Great films about cricket: Batsman Begins > hugo@REDACTED carfax.org.uk | starring Christian Bail > http://carfax.org.uk/ | > PGP: E2AB1DE4 | > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > Karolis From bob@REDACTED Tue Jan 10 22:18:55 2017 From: bob@REDACTED (Bob Ippolito) Date: Tue, 10 Jan 2017 13:18:55 -0800 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: On Tue, Jan 10, 2017 at 10:58 AM, Bhag Chandra wrote: > Hello, > > I have been coding in Erlang for 2 years. A wonderful language but not > very big community, so I cant discuss my questions with programmers around > me (Java, Python guys). I found out about this list today. > > I have some fundamental doubts about the Erlang. It would be great if > someone can help me clarify them. > > > 1) "Strings in Erlang are internally treated as a list of integers of each > character's ASCII values, this representation of string makes operations > faster. For example, string concatenation is constant time operation in > Erlang." Can someone explain why? > > The description of a string's internal representation is correct, but concatenation is O(n). Most people end up using binaries of utf8 encoded text instead of strings (lower constant factors), and iolists to concatenate data for output (which is constant time, since you're just a rose tree of data to be traversed later). > 2) "It makes sense to use Erlang only where system's availability is very > high". Is it not a very general requirement of most of the systems? > Whatsapp to Google to FB to Amazon to Paypal to Barclays etc they all are > high availability systems, so we can use Erlang in all of them? > You can use Erlang any time, but high availability in distributed systems is where Erlang is a better fit than many other options. Whatsapp in particular does use Erlang. The reasons not to use Erlang in these situations are generally related to familiarity and momentum. > 3) "Every message which is sent to a process, goes to the mailbox of that > process. When process is free, it consumes that message from mailbox". So > how exactly does process ask from the mailbox for that message? Is there a > mechanism in a process' memory which keeps polling its mailbox. I basically > want to understand how message is sent from mailbox to my code in process. > > 4) We say that a message is passed from process A to process B by simply > using a bang (!) character, but what happens behind the scenes to pass this > message? Do both processes establish a tcp connection first and then pass > message or what? > There's plenty written on these topics available in this list archive and various places online. I'd start by searching for "erlang scheduler" and then scan for the bits that talk about messages and mailboxes. TCP connections are only involved when sending messages between separate notes in distributed erlang, local messages are sent and received by manipulating local data structures. > 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) Mr. > Armstrong is talking about the difference between the context switching > overhead between OS threads and Erlang processes. He says, thread context > switching is of order 700 words but Erlang process context switching is ... > ? > I cant understand what he said, if someone could tell. > Words is a measure of memory, an Erlang process (by default) has an overhead of 309 words according to the efficiency guide. http://erlang.org/doc/efficiency_guide/processes.html -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Jan 11 01:12:01 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 11 Jan 2017 13:12:01 +1300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: On 11/01/17 7:58 AM, Bhag Chandra wrote: > 1) "Strings in Erlang are internally treated as a list of integers of > each character's ASCII values, this representation of string makes > operations faster. For example, string concatenation is constant time > operation in Erlang." Can someone explain why? Pretty much all of this is somewhere between misleading and confusing. Mind you, pretty much everything about strings in general is somewhere between confusing and misguided, whatever the programming language, especially since the introduction of Unicode. The closest thing to the truth is to say that Erlang has no native string datatype whatsoever. Instead, there are several general purpose data types that are pressed into service to fake it. (1) An atom is a unique string. There's more to say about them, but the general rule is DON'T use them for strings. (2) A list of character codes is stringy. Once upon a time character codes were 8-bit quantities, so taking two computer words per character was sneered at. Unicode has a 21-bit character set these days. (Not only that, it's now racist and sexist, so that you can distinguish between pink/yellow male/female astronaut, I kid you not. Why do we need a character "man in business suit levitating"?) The astral planes are increasingly being populated, "Version 9.0 adds exactly 7,500 characters, for a total of 128,172 characters." Let X = "1....m" and Y = "1...n". Then X ++ Y concatenates them in O(m) time. NOT O(1). (3) For many purposes, a *tree* of numbers and pairs can be used to hold text. It is *this* that allows O(1) concatenation: [X|Y] *isn't* the string "1...m1...n" but can in certain contexts *represent* it. You can, for example, build up a text of total size S in O(S) time and then flatten it (or write it or transmit it) in O(S) time. Look up the 'iolist()' type in Erlang documentation. (4) The 'binary' type was originally a byte string and is now a bit string. A byte sequence can of course represent a string, with the encoding of the text being provided by context. It used to be common to use Latin 1, it's now common to use UTF8. There is library code to help with this, but frankly, Unicode is so appallingly complex that there will probably never be enough library support (in any language). Concatenating byte strings of length m and n costs O(m+n)> (5) A tree of numbers and pairs could also contain binaries. The important thing about strings is that they are good at accepting data you don't care to inspect further, storing it, and giving it back later, and LOUSY at almost anything else, in EVERY programming language. One of the ways that they are lousy is that thanks to Unicode's rules, there is *structure* in strings (which were historically unstructured), so that you can have a well formed string of 2 characters, split it into two 1-character strings, and find that at least one of those strings is no longer legal. (Emoji flags, for example.) > > 2) "It makes sense to use Erlang only where system's availability is > very high". Is it not a very general requirement of most of the > systems? Whatsapp to Google to FB to Amazon to Paypal to Barclays etc > they all are high availability systems, so we can use Erlang in all of them? I do not think many people in this mailing list would say "it makes sense to use Erlang ONLY where availability [must be] high." People use Erlang (and other languages running on the Erlang VM, such as LFE and Elixir) for all sorts of things. Yes, Erlang is not as fast as other languages. Let's have some perspective on that, though. Just today I measured four programs to solve the same problem, using the same structure, written in four languages. C 0.04 sec -- has types Smalltalk 0.25 sec -- lacks them Java 0.65 sec -- has types Python 2.98 sec -- lacks them. Have figures like that ever stopped anyone using Python? Have they even encouraged people to switch to Smalltalk? People use a programming language when it lets them get the job at hand done to an adequate standard. These days this is as much about infrastructure such as profiling, test coverage analysis, test frameworks, documentation tools, libraries, interface support (JSON, XML, ASN.1, Protobufs, whatever) as it is about the language as such. I find that it makes sense to use Erlang whenever concurrency is important/useful, because it's so much easier to get concurrent programs right than in Java or C11 or pretty much anything else (although Ada comes close). There are of course plenty of libraries for other languages that claim to add Erlang-style abilities to those languages; what they don't mention is that you cannot remove dangers by adding libraries. For example, gets() has been deprecated in C; the OpenBSD linker tells you off if you use it. There are safer alternatives. But nothing can stop you writing your own version of gets(), and the OpenBSD linker has no idea that mygets() is just as dangerous as gets() ever was. > 3) "Every message which is sent to a process, goes to the mailbox of > that process. When process is free, it consumes that message from > mailbox". So how exactly does process ask from the mailbox for that > message? Is there a mechanism in a process' memory which keeps polling > its mailbox. I basically want to understand how message is sent from > mailbox to my code in process. "When process is free" should presumably be "When that process is RUNNING and FEELS LIKE checking its mailbox, that message MAY BE consumed if it is one the process WANTS to consume." A process reads from its mailbox using the 'receive' construction, described in for example Learn You Some Erlang For Great Good (http://learnyousomeerlang.com) which I recommend. This happens when the code decides it is time to try to receive a message. If you know about message passing in UNIX, Pid ! Msg is like msgsnd( Queue, Msg, Msg_Size, Flags) or mq_send( Queue, Msg, Msg_Size, Prio) receive ... end is like msgrcv( Queue, Msg, Msg_Size, Type, Flags) or mq_receive(Queue, Msg, Msg_Size, &Prio) That is, sending happens when there is something EXPLICIT in the code to make it happen, and receiving happens when there is something EXPLICIT in the code to make it happen, and there is buffering in between so things don't have to be simultaneous. > 4) We say that a message is passed from process A to process B by simply > using a bang (!) character, but what happens behind the scenes to pass > this message? Do both processes establish a tcp connection first and > then pass message or what? First of, we have Erlang "nodes" that are Unix or Windows "processes", and inside them we have Erlang "processes" that are like Unix or Windows "threads" except much cheaper; the word "fibres" is sometimes used to describe similar things in those environments. Erlang processes inside nodes on different machines may well use TCP connections automatically set up between the nodes. Or they could use something else. They could in principle use Infiniband or carrier pigeons. Erlang processes inside different nodes on the same machine could use any IPC facility provided by the host OS, such as System V message queues, POSIX message queues, pipes, UNIX sockets, ... Erlang processes inside the same node will use shared memory, which might or might not involve copying, depending on the Erlang version. Whatever happens, it's likely to be cheaper than TCP. > 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) Mr. > Armstrong is talking about the difference between the context switching > overhead between OS threads and Erlang processes. He says, thread > context switching is of order 700 words but Erlang process context > switching is ... ? He said that OS thread switching MOVES about 700 words, while Erlang process switching involves THREE REGISTERS. From technion@REDACTED Wed Jan 11 04:37:57 2017 From: technion@REDACTED (Technion) Date: Wed, 11 Jan 2017 03:37:57 +0000 Subject: [erlang-questions] Erlang crashes when using -name In-Reply-To: References: Message-ID: Hi, You'd be surprised how much of Windows networking breaks if you choose to disable ipv6. Microsoft's official statement on the issue: Important Internet Protocol version 6 (IPv6) is a mandatory part of Windows Vista and later versions of Windows. We do not recommend that you disable IPv6 or its components I don't know if it's related to your issue but I'm bringing it up because I've seen it be related to all manner of otherwise unexplainable issues. ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Pheasant, David T. Sent: Wednesday, 11 January 2017 3:47 AM To: erlang-questions@REDACTED Subject: [erlang-questions] Erlang crashes when using -name Hello everyone, I'm having a very weird issue with erlang that I'm hoping you all may be able to shed some light on. Long story short, we're using a 3rd party product that in turn uses RabbitMQ, which as we all know, uses Erlang. RabbitMQ will not start and I've tracked the issue down to the point where Erlang will start when a short node name is used (-sname rabbit@%computername%), but will not start with a long node name (-name rabbit@%computername%.domain.com). I've tested forward and reverse lookups of the FQDN via nslookup and the correct address is returned in all cases. I've also put the FQDN and hostname in the hosts file, which did not change the behavior. When using the long name, the erl.exe process dies immediately and does not print any error messages to the console. When it does die, Windows runs WERFault.exe, shows the familiar "erl.exe has stopped working" message, and creates the following entry in the event log: Faulting application name: erl.exe, version: 0.0.0.0, time stamp: 0x52e15856 Faulting module name: MSVCR100.dll, version: 10.0.40219.1, time stamp: 0x4d5f034a Exception code: 0xc0000005 Fault offset: 0x000000000003c120 Faulting process id: 0x1344 Faulting application start time: 0x01d26b5c79195ddd Faulting application path: C:\Program Files\ArcGIS\Server\framework\runtime\rabbitmq\erlang\bin\erl.exe Faulting module path: C:\Windows\SYSTEM32\MSVCR100.dll Report Id: b6d11c0f-d74f-11e6-80e1-00505607156a Faulting package full name: Other pertinent information: * Operating system is Windows Server 2012 * Erlang version is 5.10.4 * Host is a virtual server (vmware) * Host was migrated from on-prem to IaaS service, during which MAC address changed * IPv6 is disabled on the network adapter * Separate/isolated install of erlang 5.10.4 exhibits same behavior, so this 'feels' like a hardware/OS issue and not software configuration * When running EPMD in debug mode and using a short name, I can see erl.exe register. When using a long name, it appears that erl.exe does not attempt to register (or fails). So, my questions: * Has anyone seen this before? * If not, what differs in the startup process of erl.exe when a short name is used vs. a long name? * Could a change in MAC address have any effect on this? * What (if any) checks does erlang do when started with a long name vs short name (DNS wise)? Thanks in advance for all of your help, -David -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars.thorsen@REDACTED Wed Jan 11 07:29:39 2017 From: lars.thorsen@REDACTED (Lars Thorsen) Date: Wed, 11 Jan 2017 06:29:39 +0000 Subject: [erlang-questions] xmerl and the xml namespace In-Reply-To: References: Message-ID: Hi, this is clearly a bug and I will create an internal ticket for it. If you want to keep track on it create a bug report in Jira (http://bugs.erlang.org/). BR Lars OTP Team ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Karolis Petrauskas Sent: Saturday, January 7, 2017 10:05:31 PM To: erlang-questions@REDACTED Subject: [erlang-questions] xmerl and the xml namespace Hello erlangers, I tried to parse XML document containing namespaced elements using xmerl. The document contain xml:lang attributes and no declaration for the xml prefix. The minimal example reproducing the error is bellow. 1> catch xmerl_scan:string("", [{namespace_conformant, true}]). {'EXIT',{fatal,{{namespace_prefix_not_declared,"xml"}, {file,file_name_unknown}, {line,1}, {col,23}}}} The same XML document is parsed successfully with {namespace_conformant, false}. The XML specification (https://www.w3.org/TR/xml-names/#xmlReserved) says, that the xml prefix can be used without declaration and must refer to "http://www.w3.org/XML/1998/namespace". Is this a bug in xmerl, or I misunderstand something? Karolis _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Wed Jan 11 09:03:23 2017 From: alex0player@REDACTED (Alex S.) Date: Wed, 11 Jan 2017 11:03:23 +0300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: Not to start a holy war, but Unicode is not complex, it just has a lot of tables. And pretty much any modern language has immensely better Unicode support built in than Erlang. Thankfully there's ux. About processes and mailboxes. Every process has its share of scheduler's memory, which is used for mailbox, stack and heap. Send operation simply writes to the end of mailbox. Receive matches all the inbox (there is a temporary area too, the algorithm is clever to not block incoming messages) and selects one according to the pattern. Receiving a message which contains a freshly-created reference is optimized, otherwise it potentially has to search the whole mailbox. VM (BEAM) is actually several OS processes to allow for multi-core execution. They share almost nothing, and each one is executing one Erlang process at one time. When process switching happens, it's almost a matter of one pointer to an internal structure changing; of course, as always with virtual machines, the price is more indirection and slower code. Strings are a mess. Generally there should only be lists of characters, and binaries containing UTF-8, but some things work with lists of bytes and other such bullshit. IO lists, for one, are generally deep lists of bytes and binaries, but not always. In general, you have to be either very careful with Unicode wrangling, or just give up like many other did before you. 11 ???. 2017 ?. 3:12 ???????????? "Richard A. O'Keefe" ???????: > > > On 11/01/17 7:58 AM, Bhag Chandra wrote: > >> 1) "Strings in Erlang are internally treated as a list of integers of >> each character's ASCII values, this representation of string makes >> operations faster. For example, string concatenation is constant time >> operation in Erlang." Can someone explain why? >> > > Pretty much all of this is somewhere between misleading and confusing. > Mind you, pretty much everything about strings in general is > somewhere between confusing and misguided, whatever the programming > language, especially since the introduction of Unicode. > > The closest thing to the truth is to say that Erlang has no native > string datatype whatsoever. Instead, there are several general purpose > data types that are pressed into service to fake it. > > (1) An atom is a unique string. There's more to say about them, > but the general rule is DON'T use them for strings. > > (2) A list of character codes is stringy. Once upon a time character > codes were 8-bit quantities, so taking two computer words per character > was sneered at. Unicode has a 21-bit character set these days. (Not > only that, it's now racist and sexist, so that you can distinguish > between pink/yellow male/female astronaut, I kid you not. Why do we > need a character "man in business suit levitating"?) The astral planes > are increasingly being populated, "Version 9.0 adds exactly 7,500 > characters, for a total of 128,172 characters." > > Let X = "1....m" and Y = "1...n". > Then X ++ Y concatenates them in O(m) time. > NOT O(1). > > (3) For many purposes, a *tree* of numbers and pairs can be used to > hold text. It is *this* that allows O(1) concatenation: > [X|Y] *isn't* the string "1...m1...n" but can in certain contexts > *represent* it. You can, for example, build up a text of total > size S in O(S) time and then flatten it (or write it or transmit it) > in O(S) time. > > Look up the 'iolist()' type in Erlang documentation. > > (4) The 'binary' type was originally a byte string and is now a > bit string. A byte sequence can of course represent a string, > with the encoding of the text being provided by context. It > used to be common to use Latin 1, it's now common to use UTF8. > There is library code to help with this, but frankly, Unicode > is so appallingly complex that there will probably never be > enough library support (in any language). > > Concatenating byte strings of length m and n costs O(m+n)> > > (5) A tree of numbers and pairs could also contain binaries. > > The important thing about strings is that they are good at > accepting data you don't care to inspect further, storing it, > and giving it back later, and LOUSY at almost anything else, > in EVERY programming language. > > One of the ways that they are lousy is that thanks to Unicode's > rules, there is *structure* in strings (which were historically > unstructured), so that you can have a well formed string of > 2 characters, split it into two 1-character strings, and find > that at least one of those strings is no longer legal. > (Emoji flags, for example.) > > > >> 2) "It makes sense to use Erlang only where system's availability is >> very high". Is it not a very general requirement of most of the >> systems? Whatsapp to Google to FB to Amazon to Paypal to Barclays etc >> they all are high availability systems, so we can use Erlang in all of >> them? >> > > I do not think many people in this mailing list would say > "it makes sense to use Erlang ONLY where availability [must be] high." > > People use Erlang (and other languages running on the Erlang VM, > such as LFE and Elixir) for all sorts of things. Yes, Erlang is > not as fast as other languages. > > Let's have some perspective on that, though. > Just today I measured four programs to solve the same problem, > using the same structure, written in four languages. > C 0.04 sec -- has types > Smalltalk 0.25 sec -- lacks them > Java 0.65 sec -- has types > Python 2.98 sec -- lacks them. > Have figures like that ever stopped anyone using Python? > Have they even encouraged people to switch to Smalltalk? > > People use a programming language when it lets them get > the job at hand done to an adequate standard. > These days this is as much about infrastructure such as > profiling, test coverage analysis, test frameworks, > documentation tools, libraries, interface support (JSON, > XML, ASN.1, Protobufs, whatever) as it is about the > language as such. > > I find that it makes sense to use Erlang whenever concurrency > is important/useful, because it's so much easier to get > concurrent programs right than in Java or C11 or pretty much > anything else (although Ada comes close). There are of course > plenty of libraries for other languages that claim to add > Erlang-style abilities to those languages; what they don't > mention is that you cannot remove dangers by adding libraries. > For example, gets() has been deprecated in C; the OpenBSD > linker tells you off if you use it. There are safer alternatives. > But nothing can stop you writing your own version of gets(), > and the OpenBSD linker has no idea that mygets() is just as > dangerous as gets() ever was. > > 3) "Every message which is sent to a process, goes to the mailbox of >> that process. When process is free, it consumes that message from >> mailbox". So how exactly does process ask from the mailbox for that >> message? Is there a mechanism in a process' memory which keeps polling >> its mailbox. I basically want to understand how message is sent from >> mailbox to my code in process. >> > > "When process is free" should presumably be > "When that process is RUNNING and FEELS LIKE checking its mailbox, > that message MAY BE consumed if it is one the process WANTS to > consume." > > A process reads from its mailbox using the 'receive' construction, > described in for example Learn You Some Erlang For Great Good > (http://learnyousomeerlang.com) which I recommend. This happens > when the code decides it is time to try to receive a message. > > If you know about message passing in UNIX, > Pid ! Msg is like msgsnd( Queue, Msg, Msg_Size, Flags) > or mq_send( Queue, Msg, Msg_Size, Prio) > receive ... end is like msgrcv( Queue, Msg, Msg_Size, Type, Flags) > or mq_receive(Queue, Msg, Msg_Size, &Prio) > > That is, sending happens when there is something EXPLICIT in the > code to make it happen, and receiving happens when there is > something EXPLICIT in the code to make it happen, and there is > buffering in between so things don't have to be simultaneous. > > 4) We say that a message is passed from process A to process B by simply >> using a bang (!) character, but what happens behind the scenes to pass >> this message? Do both processes establish a tcp connection first and >> then pass message or what? >> > > First of, we have Erlang "nodes" that are Unix or Windows "processes", > and inside them we have Erlang "processes" that are like Unix or Windows > "threads" except much cheaper; the word "fibres" is sometimes used to > describe similar things in those environments. > > Erlang processes inside nodes on different machines may well use > TCP connections automatically set up between the nodes. Or they > could use something else. They could in principle use Infiniband > or carrier pigeons. > > Erlang processes inside different nodes on the same machine > could use any IPC facility provided by the host OS, such as > System V message queues, POSIX message queues, pipes, UNIX sockets, > ... > > Erlang processes inside the same node will use shared memory, > which might or might not involve copying, depending on the Erlang > version. Whatever happens, it's likely to be cheaper than TCP. > > 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) Mr. >> Armstrong is talking about the difference between the context switching >> overhead between OS threads and Erlang processes. He says, thread >> context switching is of order 700 words but Erlang process context >> switching is ... ? >> > > He said that OS thread switching MOVES about 700 words, > while Erlang process switching involves THREE REGISTERS. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Wed Jan 11 14:43:36 2017 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Wed, 11 Jan 2017 15:43:36 +0200 Subject: [erlang-questions] xmerl and the xml namespace In-Reply-To: References: Message-ID: Thanks for the information. Karolis On Wed, Jan 11, 2017 at 8:29 AM, Lars Thorsen wrote: > > Hi, > > this is clearly a bug and I will create an internal ticket for it. > > If you want to keep track on it create a bug report in Jira > (http://bugs.erlang.org/). > > > BR Lars > > OTP Team > > > ________________________________ > From: erlang-questions-bounces@REDACTED > on behalf of Karolis Petrauskas > > Sent: Saturday, January 7, 2017 10:05:31 PM > To: erlang-questions@REDACTED > Subject: [erlang-questions] xmerl and the xml namespace > > Hello erlangers, > > I tried to parse XML document containing namespaced elements using > xmerl. The document contain xml:lang attributes and no declaration for > the xml prefix. The minimal example reproducing the error is bellow. > > 1> catch xmerl_scan:string("", > [{namespace_conformant, true}]). > > {'EXIT',{fatal,{{namespace_prefix_not_declared,"xml"}, > {file,file_name_unknown}, > {line,1}, > {col,23}}}} > > The same XML document is parsed successfully with {namespace_conformant, > false}. > > The XML specification (https://www.w3.org/TR/xml-names/#xmlReserved) > says, that the xml prefix can be used without declaration and must > refer to "http://www.w3.org/XML/1998/namespace". > > Is this a bug in xmerl, or I misunderstand something? > > Karolis > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ingela.andin@REDACTED Wed Jan 11 20:12:46 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 11 Jan 2017 20:12:46 +0100 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: Hi! There is currently no supported way. ERL-166 https://bugs.erlang.org/browse/ERL-166 talks about the possibility to add such a feature. We have not had time to look further into this as yet. Of course, it is possible to provide such an API, although it seems to me that the use case is violating the concept of using TLS in the first place. It can, of course, be argued that if you have access to the erlang node you may dig out the information anyway even if it might be a dirty hack. Regards Ingela Erlang/OTP team - Ericsson AB 2017-01-06 0:13 GMT+01:00 Technion : > Hi, > > > Is it a solution to for you to deal with it on the client side? > > > https://jimshaver.net/2015/02/11/decrypting-tls-browser- > traffic-with-wireshark-the-easy-way/ > > Chrome lets you write keys out. > > > ------------------------------ > *From:* erlang-questions-bounces@REDACTED erlang.org> on behalf of Roger Lipscombe > *Sent:* Friday, 6 January 2017 1:20 AM > *To:* erlang-questions@REDACTED > *Subject:* [erlang-questions] SSL: Getting master_secret and > client_random (or premaster_secret) > > We're using ECDHE and DHE ciphers for our SSL connections. This > provides perfect forward secrecy, which is good, but it makes it > impossible to decipher packet captures in wireshark, which is > expected, and also good, almost all of the time. > > Sometimes, however, we *do* need to decipher the traffic. > > Note that we own both the client (which is embedded) and the server > (which uses Erlang -- otherwise I wouldn't be asking here -- and > ranch). We *could* offer a different cipher suite on the server, which > would disable PFS, but would do it for all connections. I'd prefer > something a bit more fine-grained. > > You can feed a key log to Wireshark, as documented at > https://developer.mozilla.org/en-US/docs/Mozilla/Projects/ > NSS/Key_Log_Format, > > > NSS Key Log Format - Mozilla | MDN > > developer.mozilla.org > Key logs can be written by NSS so that external programs can decrypt TLS > connections. Wireshark 1.6.0 and above can use these log files to decrypt > packets. > > > and it'll correctly decipher the traffic for that connection. > > I'd like to find a way to generate a key log file. This requires > either (client_random, master_secret) or (encrypted_premaster_secret, > premaster_secret). > > Note that I'm looking at the OTP 17.5 source, because that's what we're > using. > > It would seem that premaster_secret is not stored past the initial > negotiation, but the client_random and master_secret values are in the > #security_parameters record in the #connection_state record in the > #connection_states record, which is in the #state record of the SSL > connection pid. > > But I can't see any (clean) way to retrieve these values, in order to > generate a key log suitable for Wireshark. > > Is there any clean way to do this in OTP 17.5, or is there a supported > way to do this in OTP 18.x or 19.x? > > Regards, > Roger. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > erlang-questions Info Page > > erlang.org > Mailing list for general discussions about Erlang/OTP, the language, > implementation, usage, beginners questions, etc... To see the collection of > prior postings to the ... > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.des.courtis@REDACTED Wed Jan 11 17:46:09 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Wed, 11 Jan 2017 11:46:09 -0500 Subject: [erlang-questions] Future CPUs, NUMA & Erlang Message-ID: Hi Guys, I am considering fast forwarding into the future and getting a new workstation with a Knights Landing chip (a Intel? Xeon Phi? Processor 7210F or similar CPU ) from the Intel Developer Access Program for Erlang application development. This chip has 16GB on package with a data rate of over 400GB/s (100GB/s external memory) and has 64 cores (256 threads). As you can imagine this uses a NUMA architecture but it fully backwards compatible with a regular Xeon. So here is my question how ready is the Erlang VM for this type of NUMA system? What work has been done with NUMA systems and Erlang and does it apply here? Am I right to jump in now or should I wait? Cheers. Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Wed Jan 11 21:00:18 2017 From: felixgallo@REDACTED (Felix Gallo) Date: Wed, 11 Jan 2017 12:00:18 -0800 Subject: [erlang-questions] Future CPUs, NUMA & Erlang In-Reply-To: References: Message-ID: Erlang should run more than fine on those, although you may need to mess around with the NUMA topology (see the +sct flag to erl; http://erlang.org/doc/man/erl.html). Do share what you find if you take the plunge! A word of warning: many of the older Xeon Phi systems were designed for datacenters, so were rackmount, took non-standard power and had hell-banshee cooling fans that made sitting in the same room impossible. Make sure to read the label and match it to your situation before placing the request. F. On Wed, Jan 11, 2017 at 8:46 AM, Eric des Courtis < eric.des.courtis@REDACTED> wrote: > Hi Guys, > > I am considering fast forwarding into the future and getting a new > workstation with a Knights Landing chip (a Intel? Xeon Phi? Processor > 7210F or similar CPU > ) > from the Intel Developer Access Program for > Erlang application development. This chip has 16GB on package with a data > rate of over 400GB/s (100GB/s external memory) and has 64 cores (256 > threads). As you can imagine this uses a NUMA architecture but it fully > backwards compatible with a regular Xeon. > > So here is my question how ready is the Erlang VM for this type of NUMA > system? What work has been done with NUMA systems and Erlang and does it > apply here? Am I right to jump in now or should I wait? > > Cheers. > > Eric > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vans_163@REDACTED Wed Jan 11 21:02:28 2017 From: vans_163@REDACTED (Vans S) Date: Wed, 11 Jan 2017 20:02:28 +0000 (UTC) Subject: [erlang-questions] Future CPUs, NUMA & Erlang In-Reply-To: References: Message-ID: <1725803307.2058277.1484164948970@mail.yahoo.com> First result of googling "phi erlang" http://stackoverflow.com/questions/32703379/running-erlang-on-xeon-phi Perhaps learning google foo should be prioritized before you jump head first into the future. On Wednesday, January 11, 2017 2:41 PM, Eric des Courtis wrote: Hi Guys, I am considering fast forwarding into the future and getting a new workstation with a Knights Landing chip (a Intel? Xeon Phi? Processor 7210F or similar CPU) from the Intel Developer Access Program for Erlang application development. This chip has 16GB on package with a data rate of over 400GB/s (100GB/s external memory) and has 64 cores (256 threads). As you can imagine this uses a NUMA architecture but it fully backwards compatible with a regular Xeon. So here is my question how ready is the Erlang VM for this type of NUMA system? What work has been done with NUMA systems and Erlang and does it apply here? Am I right to jump in now or should I wait? Cheers. Eric _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From kennethlakin@REDACTED Wed Jan 11 22:06:38 2017 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 11 Jan 2017 13:06:38 -0800 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: <28756b76-f89d-530b-dd93-7ba049d83f64@gmail.com> On 01/11/2017 11:12 AM, Ingela Andin wrote: > Of course, it is possible to provide such an API, although it > seems to me that the use case is violating the concept of using > TLS in the first place. AFAICT, TLS protects the data in transit, provides data integrity guarantees, and provides optional peer authentication. Maybe I'm poorly-informed, but (because the peers have to perform the TLS handshaking, encryption, and decryption) it seems that it is assumed that the peers have access to the master secret, server/client random and the like. OpenSSL 1.1.0 appears to expose functions that provide the information that Lipscomb is looking for: BoringSSL (Google's fork of OpenSSL that reportedly aims to be conservative and sane) also exposes these functions: and As to extracting the information of interest in the Erlang ssl library, The two-argument variant of the ssl:connection_info function looks promising. As to implementation, it kinda looks like ssl:connection_info eventually calls ssl_connection:connection_info. One could take a cue from the code in ssl_connection:handle_call({prf ...) to extract the security_parameters from the current connection state, extract master_secret and client_random from security_parameters, and then add them to the list that ssl_connection:connection_info returns. It looks like ssl:connection_info/2 will filter out the things that weren't requested and pass the rest on to the caller. The one-argument variant returns {ConnectionProtocol, ConnectionCyphersuite}, regardless of of any additional information that the called function handed back to it. It also looks like ssl_handshake has some exported premaster_secret functions, but I'd need to do a bit of reading to figure out how to make use of them. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ok@REDACTED Wed Jan 11 22:28:16 2017 From: ok@REDACTED (ok@REDACTED) Date: Thu, 12 Jan 2017 10:28:16 +1300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: > Not to start a holy war, but Unicode is not complex, it just has a lot of > tables. Wrong. Unicode really is hugely complex. John Wheeler is quoted as saying "If you are not completely confused by quantum mechanics, you do not understand it." Now apply this to Unicode: If you think you understand Unicode, you don't. There are language tags. (This is something that was explicitly ruled out of the Unicode design, then came back in after all.) There are variation selectors. Unicode has support not just for text of varying directionality, but for *mixed* directionality, with the result that interpreting a Unicode string requires maintaining a direction stack, with characters like POP DIRECTIONAL FORMATTING. I am not saying that you have to do this yourself; what I am saying is that chopping up a Unicode string is in general not a meaningful operation. Another example: there's a set of characters like this: take next two trees and paste them horizontally take next three trees and paste them horizontally take next two trees and paste them vertically take next three trees and paste them vertically used for approximating Chinese characters not yet supported (and yes these things do nest). This means that in order to move forward one "character" in a string, it is necessary to parse a tree (no, a regular expression cannot do this; these things are *nested* and regular expressions can't do matching brackets). Did I mention the rules for emoji? Look at the rules for emoji and weep. I repeat: moving forwards or backwards ONE user-oriented character in a Unicode string is HARD. > And pretty much any modern language has immensely better Unicode > support built in than Erlang. Pretty much any modern language is moving in this area; Unicode support in modern Fortran and modern COBOL and modern C is not that great. (ICU4C is not part of any C standard.) One of the things that make Unicode difficult is that things keep *changing*. They try very hard to keep things stable, but characters do still from time to time change class (what was once an upper case letter may become a sign, for one example). I had code for moving backwards and forwards that took into account the difference between base characters and floating diacriticals. It *didn't* take variation selectors into account. (Because when I wrote the code there weren't any.) Nor did I handle language tags. (Because at the time language tags had been ruled out forever.) From g@REDACTED Wed Jan 11 22:52:37 2017 From: g@REDACTED (Garrett Smith) Date: Wed, 11 Jan 2017 15:52:37 -0600 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: On Wed, Jan 11, 2017 at 3:28 PM, wrote: >> Not to start a holy war, but Unicode is not complex, it just has a lot of >> tables. > > Wrong. Unicode really is hugely complex. > > John Wheeler is quoted as saying > "If you are not completely confused by quantum mechanics, > you do not understand it." > Now apply this to Unicode: > If you think you understand Unicode, you don't. > > There are language tags. > (This is something that was explicitly ruled out of the Unicode > design, then came back in after all.) > > There are variation selectors. > > Unicode has support not just for text of varying > directionality, but for *mixed* directionality, > with the result that interpreting a Unicode string > requires maintaining a direction stack, with characters > like POP DIRECTIONAL FORMATTING. > > I am not saying that you have to do this yourself; > what I am saying is that chopping up a Unicode string > is in general not a meaningful operation. > > Another example: there's a set of characters like this: > take next two trees and paste them horizontally > take next three trees and paste them horizontally > take next two trees and paste them vertically > take next three trees and paste them vertically > used for approximating Chinese characters not yet > supported (and yes these things do nest). > > This means that in order to move forward one "character" > in a string, it is necessary to parse a tree (no, a > regular expression cannot do this; these things are > *nested* and regular expressions can't do matching > brackets). > > Did I mention the rules for emoji? Look at the rules > for emoji and weep. > > I repeat: moving forwards or backwards ONE > user-oriented character in a Unicode string is HARD. > >> And pretty much any modern language has immensely better Unicode >> support built in than Erlang. > > Pretty much any modern language is moving in this area; > Unicode support in modern Fortran and modern COBOL and > modern C is not that great. (ICU4C is not part of any > C standard.) > > One of the things that make Unicode difficult is that things > keep *changing*. They try very hard to keep things stable, > but characters do still from time to time change class > (what was once an upper case letter may become a sign, for > one example). > > I had code for moving backwards and forwards that took into > account the difference between base characters and floating > diacriticals. It *didn't* take variation selectors into account. > (Because when I wrote the code there weren't any.) > Nor did I handle language tags. > (Because at the time language tags had been ruled out forever.) I am completely confused by this AND ALSO I do not understand it. From nbartley@REDACTED Wed Jan 11 23:30:56 2017 From: nbartley@REDACTED (nato) Date: Wed, 11 Jan 2017 14:30:56 -0800 Subject: [erlang-questions] Erlang shell, process dictionary & rand:seed Message-ID: In the Erlang shell, I noticed that `rand:seed/n` set my shell process dictionary value with the seed (looked at it with `process_info(self())`). When I crashed my shell process with `1 / 0`, my new shell process inherited the old process's process dictionary value (those seed integers). Is this expected? Is this just an aid for just the shell? And if not, what can I expect for non-shell processes with the above interplay? From eric.des.courtis@REDACTED Wed Jan 11 22:22:31 2017 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Wed, 11 Jan 2017 16:22:31 -0500 Subject: [erlang-questions] Future CPUs, NUMA & Erlang In-Reply-To: <1725803307.2058277.1484164948970@mail.yahoo.com> References: <1725803307.2058277.1484164948970@mail.yahoo.com> Message-ID: Vans, I already looked into this. The KNL generation are not co-processors like the older generation Xeon Phi which had some ISA compatibility issues. Theses chips are 100% compatible with ordinary Xeons. Eric On Wed, Jan 11, 2017 at 3:02 PM, Vans S wrote: > First result of googling "phi erlang" > http://stackoverflow.com/questions/32703379/running-erlang-on-xeon-phi > Perhaps learning google foo should be prioritized before you jump head > first into the future. > > > On Wednesday, January 11, 2017 2:41 PM, Eric des Courtis < > eric.des.courtis@REDACTED> wrote: > > > > Hi Guys, > > I am considering fast forwarding into the future and getting a new > workstation with a Knights Landing chip (a Intel? Xeon Phi? Processor 7210F > or similar CPU) from the Intel Developer Access Program for Erlang > application development. This chip has 16GB on package with a data rate of > over 400GB/s (100GB/s external memory) and has 64 cores (256 threads). As > you can imagine this uses a NUMA architecture but it fully backwards > compatible with a regular Xeon. > > So here is my question how ready is the Erlang VM for this type of NUMA > system? What work has been done with NUMA systems and Erlang and does it > apply here? Am I right to jump in now or should I wait? > > Cheers. > > Eric > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Thu Jan 12 00:09:18 2017 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 11 Jan 2017 23:09:18 +0000 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: <28756b76-f89d-530b-dd93-7ba049d83f64@gmail.com> References: <28756b76-f89d-530b-dd93-7ba049d83f64@gmail.com> Message-ID: On 11 January 2017 at 21:06, Kenneth Lakin wrote: > As to extracting the information of interest in the Erlang ssl library, > The two-argument variant of the ssl:connection_info function looks > promising. I've not looked at the code, but that sounds like a sensible place to graft this in, sure. > It also looks like ssl_handshake has some exported premaster_secret > functions, but I'd need to do a bit of reading to figure out how to make > use of them. I'm not 100% sure, but I believe that the functions in ssl_handshake are used to generate premaster_secret and, from that, the master_secret, etc. I only had a cursory glance, though. Cheers, Roger. From roger@REDACTED Thu Jan 12 00:17:06 2017 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 11 Jan 2017 23:17:06 +0000 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: On 11 January 2017 at 19:12, Ingela Andin wrote: > There is currently no supported way. ERL-166 https://bugs.erlang.org/ > browse/ERL-166 talks about the possibility to add such a feature. We have > not had time to look further into this as yet. > I'm happy to submit a PR to implement this, provided we can agree on the approach (but it'll be a month or two -- we're still on Erlang 17.x, and there's no point in submitting a patch against that). > Of course, it is possible to provide such an API, although it seems to me > that the use case is violating the concept of using TLS in the first place. > It can, of course, be argued that if you have access to the erlang node you > may dig out the information anyway even if it might be a dirty hack. > I *would* argue that: We own the server, so the unencrypted traffic is already available. All this is doing is making it easier to see that data in wireshark, where there's a bunch of other useful context. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Thu Jan 12 01:40:07 2017 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 11 Jan 2017 16:40:07 -0800 Subject: [erlang-questions] [ANN] CloudI 1.6.0 Released! Message-ID: Download 1.6.0 from http://sourceforge.net/projects/cloudi/files/latest/download (checksums at the bottom of this email) CloudI (http://cloudi.org/) is a "universal integrator" using an Erlang core to provide fault-tolerance with efficiency and scalability. The CloudI API provides a minimal interface to communicate among services so programming language agnostic and protocol agnostic integration can occur. CloudI currently integrates with the programming languages C/C++, Elixir, Erlang, Java, JavaScript/node.js, PHP, Perl, Python, and Ruby, while including many reusable services that rely on the CloudI service bus. The details for this release are below: * backwards compatibility difference: * Functions deprecated in the 1.3.3 CloudI release have been removed * Usage of the 'immediate' atom for the timeout value with the cloudi_service Erlang module and the cloudi Erlang module is deprecated, use the 'limit_min' atom instead for the same value * The minimum allowable values for service configuration (both adding a new service and updating an existing service) of timeout_async and timeout_sync has changed from 101 milliseconds to 499 milliseconds to be consistent with the value that 'limit_min' represents * The service configuration options request_timeout_immediate_max and response_timeout_immediate_max had their meaning change to rely on a greater than or equal to comparison instead of a greater than comparison, though the same default behavior is preserved with this change (http://cloudi.org/api.html#2_services_add_config_opts) * Now cloudi_core doesn't use any port drivers or NIFs except what is provided within Erlang/OTP, and process dictionary use is minimal * The Elixir module CloudILogger is now included in the cloudi_core Erlang application to simplify usage of CloudI in Elixir source code * Bugs related to CloudI logging syslog integration have been fixed and more options for syslog use have been added (http://cloudi.org/api.html#2_logging_syslog_set has more information) * The restart_delay service configuration option was added (http://cloudi.org/api.html#2_services_add_config_opts_restart_delay) * CloudI service restarts have been made synchronous to ensure all termination has completed before initialization of new processes begins (to improve reliable management of global state during a restart) * All service configuration timeout values that take milliseconds also accept the Erlang atoms 'limit_min' and 'limit_max' where 'limit_max' is a concept similar to the 'infinity' atom in Erlang source code, though it does represent 49.7 days in milliseconds, so be careful about its use (no one wants to wait 49.7 days to see a failure) * All CloudI API implementations now handle a null return value as a response to create a null response, if the programming language has a concept of null (for more information, see the null integration test http://cloudi.org/tutorials.html#cloudi_examples and/or read about a null response at http://cloudi.org/faq.html#4_Null) * All CloudI API implementations now put stdout and stderr into unbuffered mode to ensure CloudI service output is logged as quickly as possible * Bugs were fixed and other improvements were added (see the ChangeLog for more detail) Please mention any problems, issues, or ideas! Thanks, Michael SHA256 CHECKSUMS cloudi-1.6.0.tar.gz (14363598 bytes) 4f14f508a2a4840026faf38c8abd7b13f314c619a8bafd59703cf59b1dd864a1 cloudi-1.6.0.tar.bz2 (12330051 bytes) 1045cceb890fa7310c80aa6e783c5b480e1700346ff6d26cac67dbbd35704518 From ok@REDACTED Thu Jan 12 01:52:01 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 12 Jan 2017 13:52:01 +1300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: On 12/01/17 10:52 AM, Garrett Smith wrote: > I am completely confused by this AND ALSO I do not understand it. Join the club! Seriously, I recommend the book "Unicode Demystified". On the one hand, it cleared up a lot of things I thought I understood but didn't. On the other hand, it revealed large complex aspects of Unicode that I was too ignorant of to know how ignorant I was. On the gripping hand, it is several major releases out of date, and things have only got worse. But it *is* a great foundation. From steven.charles.davis@REDACTED Thu Jan 12 02:50:38 2017 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 11 Jan 2017 19:50:38 -0600 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead Message-ID: You would think that, by now, computer science would have a decent answer to how to deal with text. All answers (from anyplace anyblog anylist) I have heard so far basically? suck. Why is that? What are the missing ingredients to a real solution? The word ?string? for me has long been a dirty word. /s (Political aside: *spits on all that ?emoji? crap* :-) ) From raimo+erlang-questions@REDACTED Thu Jan 12 09:41:35 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 12 Jan 2017 09:41:35 +0100 Subject: [erlang-questions] Erlang shell, process dictionary & rand:seed In-Reply-To: References: Message-ID: <20170112084135.GA12941@erix.ericsson.se> On Wed, Jan 11, 2017 at 02:30:56PM -0800, nato wrote: > In the Erlang shell, I noticed that `rand:seed/n` set my > shell process dictionary value with the seed > (looked at it with `process_info(self())`). > > When I crashed my shell process with `1 / 0`, my new > shell process inherited the old process's process dictionary > value (those seed integers). > > Is this expected? Is this just an aid for just the shell? And if not, > what can I expect for non-shell processes with the above > interplay? It is a shell feature. Notice below - an "** exception error:" is different from "*** Shell process terminated! ***". 1/0 is as well as most other exceptions is caught by the evaluator so the process will not exit. Erlang/OTP 17 [erts-6.4.1.2] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V6.4.1.2 (abort with ^G) 1> 1/0. ** exception error: an error occurred when evaluating an arithmetic expression in operator '/'/2 called as 1 / 0 2> exit(). *** Shell process terminated! *** Eshell V6.4.1.2 (abort with ^G) 1> put(a, 1). undefined 2> get(). [{a,1}] 3> 1/0. ** exception error: an error occurred when evaluating an arithmetic expression in operator '/'/2 called as 1 / 0 4> get(). [{a,1}] 5> exit(). *** Shell process terminated! *** Eshell V6.4.1.2 (abort with ^G) 1> get(). [] 2> -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From roger@REDACTED Thu Jan 12 10:58:06 2017 From: roger@REDACTED (Roger Lipscombe) Date: Thu, 12 Jan 2017 09:58:06 +0000 Subject: [erlang-questions] Erlang shell, process dictionary & rand:seed In-Reply-To: <20170112084135.GA12941@erix.ericsson.se> References: <20170112084135.GA12941@erix.ericsson.se> Message-ID: On 12 January 2017 at 08:41, Raimo Niskanen wrote: > It is a shell feature. > > Notice below - an "** exception error:" is different from > "*** Shell process terminated! ***". 1/0 is as well as most other > exceptions is caught by the evaluator so the process will not exit. Except that the process *does* exit (note that my shell displays the pid in the prompt, but I've added self() calls to be clear): Erlang/OTP 17 [erts-6.4.1.2] [source-73fc7ca] [64-bit] [smp:12:12] [async-threads:10] [hipe] [kernel-poll:false] Eshell V6.4.1.2 (abort with ^G) <0.33.0> 1> self(). <0.33.0> <0.33.0> 2> put(a, 1). undefined <0.33.0> 3> get(). [{a,1}] <0.33.0> 4> 1/0. ** exception error: an error occurred when evaluating an arithmetic expression in operator '/'/2 called as 1 / 0 <0.38.0> 5> self(). <0.38.0> <0.38.0> 6> get(). [{a,1}] From raimo+erlang-questions@REDACTED Thu Jan 12 13:43:52 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 12 Jan 2017 13:43:52 +0100 Subject: [erlang-questions] Erlang shell, process dictionary & rand:seed In-Reply-To: References: <20170112084135.GA12941@erix.ericsson.se> Message-ID: <20170112124352.GA35077@erix.ericsson.se> On Thu, Jan 12, 2017 at 09:58:06AM +0000, Roger Lipscombe wrote: > On 12 January 2017 at 08:41, Raimo Niskanen > wrote: > > It is a shell feature. > > > > Notice below - an "** exception error:" is different from > > "*** Shell process terminated! ***". 1/0 is as well as most other > > exceptions is caught by the evaluator so the process will not exit. > > Except that the process *does* exit (note that my shell displays the > pid in the prompt, but I've added self() calls to be clear): > > Erlang/OTP 17 [erts-6.4.1.2] [source-73fc7ca] [64-bit] [smp:12:12] > [async-threads:10] [hipe] [kernel-poll:false] > > Eshell V6.4.1.2 (abort with ^G) > <0.33.0> 1> self(). > <0.33.0> > <0.33.0> 2> put(a, 1). > undefined > <0.33.0> 3> get(). > [{a,1}] > <0.33.0> 4> 1/0. > ** exception error: an error occurred when evaluating an arithmetic expression > in operator '/'/2 > called as 1 / 0 > <0.38.0> 5> self(). > <0.38.0> > <0.38.0> 6> get(). > [{a,1}] Well, how about that!? The shell does strange things. It is for example divided into two processes; one evaluator process and one other process that I think for instance owns links, receives messages, etc. It seems the shell emulates what I said i.e that an exception was caught, but the process receiving messages actually died due to the exception. Confusing. I wonder if it is possible to correct...? Nevertheless, if a non-emulated process dies due to a runtime error e.g division by zero, its process dictionary, links and other process stuff dies with it. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From mbike2000ru@REDACTED Thu Jan 12 13:50:01 2017 From: mbike2000ru@REDACTED (Dmitry) Date: Thu, 12 Jan 2017 12:50:01 +0000 (UTC) Subject: [erlang-questions] timer:sleep counts incorrectly inside my application References: <710577707.17058.1484225401431.ref@mail.yahoo.com> Message-ID: <710577707.17058.1484225401431@mail.yahoo.com> Hello It counts perfectly in an erlang console but here I need to send 70 push tokens with an interval 1 minute - but it sends with intervals 6-10 seconds send_it(_A, BinToken) ->? ? timer:sleep(60000),? ? lager:info("The BinToken is in Send_it: ~p ~n",[BinToken]), ? ?Body = "{\"app_id\": \"c35654655556445454550a6\",\"contents\": {\"en\": \"English Message\"}, \"include_player_ids\": [\"" ++ binary_to_list(BinToken) ++"\"]}", ? ?Request = {"https://onesignal.com/api/v1/notifications",? ? ? ? ? ? ? [{"Authorization","Basic YTM25464464546456565QyZDlh"}], "application/json", Body}, ? ?httpc:request(post, Request, [], []). annoy (BinToken) ->? [send_it(A, BinToken) || A <- lists:seq(1,70)].++++++++++++++++++++++++++++++++++++++++++++++++++++where to dig? -------------- next part -------------- An HTML attachment was scrubbed... URL: From vances@REDACTED Thu Jan 12 14:27:41 2017 From: vances@REDACTED (Vance Shipley) Date: Thu, 12 Jan 2017 18:57:41 +0530 Subject: [erlang-questions] Refactoring for edoc Message-ID: We've recently discovered that edoc generates much nicer documentation when you write type specifications using `when` such as: -spec example(Number) -> {ok, Result} | {error, Reason} when Number :: pos_integer(), Result = integer(), Reason :: term(). ... rather than the more obvious: -spec example(Number :: pos_integer()) -> {ok, Result :: integer()} | {error, Reason :: term()}. So the question becomes how can we easily refactor all of our type specifications to use the former method? Is there a refactoring too well suited to this job? -- -Vance From ilya.khaprov@REDACTED Thu Jan 12 14:36:24 2017 From: ilya.khaprov@REDACTED (Ilya Khaprov) Date: Thu, 12 Jan 2017 13:36:24 +0000 Subject: [erlang-questions] Refactoring for edoc In-Reply-To: References: Message-ID: Hello Why it generates different docs anyway? Regards, Ilya From: Vance Shipley Sent: Thursday, January 12, 2017 04:27 PM To: erlang-questions Subject: [erlang-questions] Refactoring for edoc We've recently discovered that edoc generates much nicer documentation when you write type specifications using `when` such as: -spec example(Number) -> {ok, Result} | {error, Reason} when Number :: pos_integer(), Result = integer(), Reason :: term(). ... rather than the more obvious: -spec example(Number :: pos_integer()) -> {ok, Result :: integer()} | {error, Reason :: term()}. So the question becomes how can we easily refactor all of our type specifications to use the former method? Is there a refactoring too well suited to this job? -- -Vance _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Thu Jan 12 15:00:04 2017 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Thu, 12 Jan 2017 15:00:04 +0100 Subject: [erlang-questions] Refactoring for edoc In-Reply-To: References: Message-ID: Instead of rewriting everyone's code, my vote is on adding an option to Edoc that would pretty print specs as if they were written using `when`. Here is a mix of the two styles, as an example: http://erldocs.com/current/edoc/edoc_lib.html?i=5&search=edoc#edoc Note to self: fix that bug where instead of `[term()]` it should say `proplist() = [term()]`! Cheers, -- Pierre Fenoll On 12 January 2017 at 14:36, Ilya Khaprov wrote: > Hello > > > > Why it generates different docs anyway? > > > > Regards, > > Ilya > > > > *From: *Vance Shipley > *Sent: *Thursday, January 12, 2017 04:27 PM > *To: *erlang-questions > *Subject: *[erlang-questions] Refactoring for edoc > > > We've recently discovered that edoc generates much nicer documentation > when you write type specifications using `when` such as: > > -spec example(Number) -> {ok, Result} | {error, Reason} > when Number :: pos_integer(), Result = integer(), Reason :: > term(). > > ... rather than the more obvious: > > -spec example(Number :: pos_integer()) -> {ok, Result :: > integer()} | {error, Reason :: term()}. > > > So the question becomes how can we easily refactor all of our type > specifications to use the former method? Is there a refactoring too > well suited to this job? > > -- > -Vance > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From silviu.cpp@REDACTED Thu Jan 12 16:30:30 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Thu, 12 Jan 2017 17:30:30 +0200 Subject: [erlang-questions] enif_consume_timeslice performances Message-ID: Hello, I'm working on a TLS/DTLS library in top of BoringSSL/OpenSSL and gen_tcp. My goal is to have a interface compatible as much as I can with erlang ssl module. Project is located here: https://github.com/silviucpp/erltls and overall so far I have better performances results than ssl module. around 33 % on OSX and 18 % on ubuntu when using ssl as client for sending the requests (when using erltls as client boost is even more for example on mac around 60 %) To measure the performances I'm using : https://github.com/silviucpp/tls_bench During testing I observed that when using enif_consume_timeslice I have a drop off in performances for around 15 %. It's calling this method too heavy ? or the processes are suspended/resumed to many times and this affects the results? Silviu -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Jan 12 20:56:38 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 12 Jan 2017 19:56:38 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: Strings are really just streams of bytes. Streams of bytes are the ubiquitous data representation format. It can represent anything you like. The problem is that with the vast generality comes lack of precision. In order to handle the string, you need to interpret its contents. You that with a spectrum of string manipulation techniques: splitting, converting to other types, regular expression matching, LALR(1) parsing and so on. The goal is to take the stream of bytes into some deeper structure which you can manipulate in the program. Squint your eyes enough, and all programs are string processing. Once you start realizing everything is basically a variant of string processing, you see why it must be hard to build a good solution for. Your solution is at some end of the spectrum, which makes it excellent for handling some problems, but makes it hard to handle others. Very general solutions to the processing problem (LALR(1) parsers, etc) are not the easiest to get going on smaller problems. Hence most processing is complicated because it requires you to pick the right kind of data processing abstraction in addition to solving the problem itself. Natural text is hard because there is very little formal structure in it. So you have to use approximative methods, Machine Learning, etc. Context Free text is easier to manage, but few people do it correctly and write a parser for it. Rather, they tend to try to work directly on the string (jokingly called "stringly typed programming"). Is Erlang bad at string processing? Yes and no. OCaml processes strings much faster than Erlang in the common case because it is a compiled language and can run directly as machine code. OTOH, if you have to look at every byte in a data stream, chances are you are bound not by the CPU time, but the time it takes to get data close to the CPU from DRAM. As for the API availability, it all depends on how good you are at using functional programming I guess. I rarely feel I lack certain API functions in Erlang when handling strings. I'd be keen to look into woes you've had in the past with things that were unwieldy to handle. On Thu, Jan 12, 2017 at 2:50 AM Steve Davis wrote: > > You would think that, by now, computer science would have a decent answer > to how to deal with text. > > All answers (from anyplace anyblog anylist) I have heard so far basically? > suck. > > Why is that? What are the missing ingredients to a real solution? > > The word ?string? for me has long been a dirty word. > > /s > > (Political aside: *spits on all that ?emoji? crap* :-) ) > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Jan 13 02:33:56 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 13 Jan 2017 14:33:56 +1300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: > Strings are really just streams of bytes. That was true a long time ago. Maybe. But it isn't anywhere near accurate as a description of Unicode: - Unicode is made of 21-bit code points, not bytes. - Most possible code points are not defined. - Some of those that are defined are defined as "it is illegal to use this". - Unicode sequences have *structure*; it is simply not the case that every sequence of allowable Unicode code points is a legal Unicode string. - As a special case of that, if s is a non-empty valid Unicode string, it is not true that every substring of s is a valid Unicode string. In case you were thinking of UTF-8, not all byte sequences are valid UTF-8. Byte streams are as important as you say, but it's really hard to see the software for a radar or a radio telescope as processing strings... From erlang@REDACTED Fri Jan 13 10:00:48 2017 From: erlang@REDACTED (Erik Reitsma) Date: Fri, 13 Jan 2017 10:00:48 +0100 Subject: [erlang-questions] FOSDEM 2017 anyone? Message-ID: Hi all! Anyone else going to FOSDEM 2017 in Brussels? I did not see any Erlang talks on this year's schedule, but perhaps some other Erlang users would like to meet up there and have a chat about our favorite (?) language? Regards, *Erik. From ingela.andin@REDACTED Fri Jan 13 10:39:55 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 13 Jan 2017 10:39:55 +0100 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: Hi! 2017-01-12 0:17 GMT+01:00 Roger Lipscombe : > On 11 January 2017 at 19:12, Ingela Andin wrote: > >> There is currently no supported way. ERL-166 >> https://bugs.erlang.org/browse/ERL-166 talks about the possibility to >> add such a feature. We have not had time to look further into this as yet. >> > > I'm happy to submit a PR to implement this, provided we can agree on the > approach (but it'll be a month or two -- we're still on Erlang 17.x, and > there's no point in submitting a patch against that). > > >> Of course, it is possible to provide such an API, although it seems to me >> that the use case is violating the concept of using TLS in the first place. >> It can, of course, be argued that if you have access to the erlang node you >> may dig out the information anyway even if it might be a dirty hack. >> > > I *would* argue that: We own the server, so the unencrypted traffic is > already available. All this is doing is making it easier to see that data > in wireshark, where there's a bunch of other useful context. > Well our reasoning at the moment is that we could add a debug possibility, that would let connection_information (ssl connection_info is deprecated due to its inflexible return value from old ssl). return client/server/master_secret values for connections started in debug mode. Just like you can configure a connection to run anonymous ciphers suites for test and debugging purposes. However we would not want connection_information to return these values by default. Even if you conceptually can get at the information by hacking we do not want to make it easy to do bad things to security by "accident" or otherwise. Regards Ingela Erlang/OTP team - Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Fri Jan 13 11:43:45 2017 From: roger@REDACTED (Roger Lipscombe) Date: Fri, 13 Jan 2017 10:43:45 +0000 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: On 13 January 2017 at 09:39, Ingela Andin wrote: > Well our reasoning at the moment is that we could add a debug possibility, > that would let connection_information > return client/server/master_secret values for connections started in debug > mode. Just like you can configure a connection to run anonymous ciphers > suites for test and debugging purposes. However we would > not want connection_information to return these values by default. Even if > you conceptually can get at the information by hacking we do not want to > make it easy to do bad things to security by "accident" or > otherwise. Sure. There's precedent for that: process_info/1 doesn't return everything that you can ask for in process_info/2, no? I'm not sure how this would do bad things to security. The server's already seeing the plaintext, otherwise it couldn't do its job. Could you explain your concerns further? From jesper.louis.andersen@REDACTED Fri Jan 13 13:20:05 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 13 Jan 2017 12:20:05 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: Richard is indeed right, depending on what your definition of "String" is. If a "String" is "An array of characters from some alphabet", then you need to take into account Strings are Unicode codepoints in practice. This is also the most precise definition from a technical point of view. When I wrote my post, I was--probably incorrectly--assuming the older notion of a "String" where the representation is either ASCII or something like ISO-8859-15. In this case, a string coincides with a stream of bytes. Data needs parsing. A lot of data comes in as some kind of stringy representation: UTF-8, byte array (binary), and so on. And of course, that isn't the whole story, since there are examples of input which are not string-like in their forms. On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe wrote: > > > On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: > > Strings are really just streams of bytes. > > That was true a long time ago. Maybe. > But it isn't anywhere near accurate as a description > of Unicode: > - Unicode is made of 21-bit code points, not bytes. > - Most possible code points are not defined. > - Some of those that are defined are defined as > "it is illegal to use this". > - Unicode sequences have *structure*; it is simply > not the case that every sequence of allowable > Unicode code points is a legal Unicode string. > - As a special case of that, if s is a non-empty > valid Unicode string, it is not true that every > substring of s is a valid Unicode string. > > In case you were thinking of UTF-8, not all byte > sequences are valid UTF-8. > > Byte streams are as important as you say, but it's > really hard to see the software for a radar or a > radio telescope as processing strings... > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.orendorff@REDACTED Thu Jan 12 20:21:57 2017 From: jason.orendorff@REDACTED (Jason Orendorff) Date: Thu, 12 Jan 2017 13:21:57 -0600 Subject: [erlang-questions] enif_send rules questions Message-ID: Hi all, The documentation for enif_send says this: env The environment of the calling process. Must be NULL only if calling from a created thread. I have a few questions about this. 1. What is "the environment of the calling process"? Does that mean the environment that was passed to the NIF? 2. What is a "created thread"? 3. Is it really invalid to do this from a NIF call -- unless I spawn a separate thread to do it, and then it's OK? NIF_ENV msg_env = enif_alloc_env(); NIF_TERM msg = enif_make_int(env, 1); enif_send(NULL, &pid, msg_env, msg); enif_free_env(msg_env); 4. We suspect it's actually totally safe to pass a process-independent environment as the first parameter, regardless of whether there's a "calling process" or we're "calling from a created thread". Is it? If so, would you accept a patch to document that that's OK? Background: I'm working on Rustler, a library for writing Erlang NIFs in the Rust programming language. Rustler aims to be safe: Rustler NIFs can't trigger undefined behavior or crash the VM, as long as you avoid Rust's `unsafe` keyword. If you manage to crash, it's not your bug. It's ours. This means Rustler must enforce all "erl_nif.h" API usage rules. We can enforce most rules with zero overhead. \o/ In a few cases, we add a runtime check. But `enif_send` is different. We have to know whether or not we're in a "created thread". If not, we have to know "the environment of the calling process", because passing any other environment would be illegal. That imposes a bit of overhead on every NIF call -- on entry, we have to stash the env in thread-local storage. I measured, and on Linux, it costs about 133ns per NIF call. Not awful, but we'd have to impose that on everyone, whether they use `enif_send` or not. So I thought I'd ask. The changes proposed in question #4 would eliminate the need for this overhead, because all environments would always be OK for sending. -- Thanks for reading! Jason Orendorff -------------- next part -------------- An HTML attachment was scrubbed... URL: From himars@REDACTED Fri Jan 13 10:05:10 2017 From: himars@REDACTED (Jack Tang) Date: Fri, 13 Jan 2017 17:05:10 +0800 Subject: [erlang-questions] How can I release beam memory? Message-ID: Hello list, I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command. ``` KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers KiB Swap: 0 total, 0 used, 0 free. 18684864 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp ``` I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow: ``` lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children(some_thing_sup)). ``` and erlang:memory() decreases from 32G to 23G. ``` [{total,22982011544}, {processes,12884182336}, {processes_used,12884170336}, {system,10097829208}, {atom,13828705}, {atom_used,13796692}, {binary,170530288}, {code,16450626}, {ets,9637717576}] ``` However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks BR -- Jack Tang http://www.linkedin.com/in/jacktang -------------- next part -------------- An HTML attachment was scrubbed... URL: From karolis.velicka@REDACTED Fri Jan 13 11:49:31 2017 From: karolis.velicka@REDACTED (Karl Velicka) Date: Fri, 13 Jan 2017 10:49:31 +0000 Subject: [erlang-questions] FOSDEM 2017 anyone? In-Reply-To: References: Message-ID: Hello! I am attending, and would be interested in meeting some Erlang users! And indeed, it's a shame that there don't seem to be any Erlang related talks.. All the best, Karl On 13 Jan 2017 09:01, "Erik Reitsma" wrote: Hi all! Anyone else going to FOSDEM 2017 in Brussels? I did not see any Erlang talks on this year's schedule, but perhaps some other Erlang users would like to meet up there and have a chat about our favorite (?) language? Regards, *Erik. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Fri Jan 13 14:10:29 2017 From: steven.charles.davis@REDACTED (Steve Davis) Date: Fri, 13 Jan 2017 07:10:29 -0600 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: <0B44A6BE-EC31-4DD1-8677-3D8FEF567043@gmail.com> > On Jan 13, 2017, at 6:20 AM, Jesper Louis Andersen wrote: > > depending on what your definition of "String" is ..and that?s the problem with strings. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri Jan 13 15:14:34 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 13 Jan 2017 15:14:34 +0100 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: Hi! 2017-01-13 11:43 GMT+01:00 Roger Lipscombe : > On 13 January 2017 at 09:39, Ingela Andin wrote: > > Well our reasoning at the moment is that we could add a debug > possibility, > > that would let connection_information > > return client/server/master_secret values for connections started in > debug > > mode. Just like you can configure a connection to run anonymous ciphers > > suites for test and debugging purposes. However we would > > not want connection_information to return these values by default. Even > if > > you conceptually can get at the information by hacking we do not want to > > make it easy to do bad things to security by "accident" or > > otherwise. > > Sure. There's precedent for that: process_info/1 doesn't return > everything that you can ask for in process_info/2, no? > > I'm not sure how this would do bad things to security. The server's > already seeing the plaintext, otherwise it couldn't do its job. Could > you explain your concerns further? > As long as it stays on the server.... TLS is suppose to provide peer to peer security and you are not suppose to be able to read TLS data in traffic sniffing logs. What if someone decides to transfer the logs in an insecure way from the server! What if someone thinks its a good idea to decrypt the data outside the TLS connection in the server and send it to an external logging server in the clear! Openssl also warns for missuse: https://www.openssl.org/docs/man1.1.0/ssl/SSL_SESSION_get_master_key.html When it comes to security you should be very careful is all I am saying, and providing a way for others to use secret information in a not intended way is potentially dangerous. Regards Ingela Erlang/OTP Team - Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.reix@REDACTED Fri Jan 13 15:24:18 2017 From: tony.reix@REDACTED (REIX, Tony) Date: Fri, 13 Jan 2017 14:24:18 +0000 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> Message-ID: <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> Hi, I'd like to port Erlang/OTP on AIX. After browsing several files and ReadMe files in the source and on Erlang web-site, it seems to me that AIX is not yet handled by Erlang. I've already started to look at the code. And it appears that there is some assembler code for PowerPC, probably for Linux. However, browsing the code, I see that it talks of ELF (AIX is XCOFF) and of AIX. I'm quite lost. Moreover, I've read that only 32bit Linux/PowerPC is handled. No 64bits. Correct ? I'd like to find information about: - which parts must be modified/extended for handling AIX (XCOFF) ? - which parts must be modified/extended for handling 64bit in addition to PowerPC 32bit ? Any other information about where I have to look at for porting Erlang (v 19.2 now) on AIX would be very useful for me. Thanks Regards Tony Reix ATOS/Bull company Grenoble - FRANCE -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Fri Jan 13 15:35:31 2017 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 13 Jan 2017 15:35:31 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <0B44A6BE-EC31-4DD1-8677-3D8FEF567043@gmail.com> References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> <0B44A6BE-EC31-4DD1-8677-3D8FEF567043@gmail.com> Message-ID: When I teach programming I always talk about the distinction between 'strings' and string literals. Think that the two are the same is a big mistake. In Erlang ' "aaa" ' is a 'string literal' and NOT a string. String literals in most programming languages are sequences of characters interposed between ' " ' characters. If I say (in Erlang) X = "aaaa" Then to the right hand side of the equals we have a string literal. The parser turns this into some internal form which is stored in the variable X - thereafter we can speak of value of X as being a "string", it's actually defined to be a list of integers, where each integer represents a single character. In C x = "aaaa" the internal representation is *entirely different* - this gets turned into a zero terminated sequence of chas. Most (but not all) C compilers store chars in 8 bit bytes so if your natural language has more than 256 characters problems occur. Enter UTF8 and friends - these are just smart ways of encoding large integers (ie characters > 256) into multi byte sequences. Then we have Unicode which has mappings from natural language character sets onto integers and complex rules for overlaying and flowing characters. Erlang takes the view that the internal representation of a string literal should be a list of integers - and since Erlang integers are bignums then any character in any language can be represented as a single integer in such a list - so far so good. The problem now comes when we want to convert from a natural language text into an Erlang string, and for manipulating strings, and for this a number of libraries are available. People complain about string handling in Erlang, but these complaints usually seem to reflect the fact that the libraries that manipulate strings and the escape conventions used in string literals differ from those in other programming languages. String handling in all languages is one unholy mess, where the stupid conventions in one language are copied by later languages. For example, English has a start-quote and end-quote symbol and they *differ*. "This editor," I think, "gets quotes wrong" [1] - amazing - how many programmers has Google got? - and they can't get quotes right. Text manipulation on a computer should be far easier than it is. Cheers /Joe [1] Google's Chrome browser On Fri, Jan 13, 2017 at 2:10 PM, Steve Davis wrote: > > On Jan 13, 2017, at 6:20 AM, Jesper Louis Andersen > wrote: > > depending on what your definition of "String" is > > > ..and that?s the problem with strings. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From rvirding@REDACTED Fri Jan 13 15:55:54 2017 From: rvirding@REDACTED (Robert Virding) Date: Fri, 13 Jan 2017 15:55:54 +0100 Subject: [erlang-questions] Erlang shell, process dictionary & rand:seed In-Reply-To: <20170112124352.GA35077@erix.ericsson.se> References: <20170112084135.GA12941@erix.ericsson.se> <20170112124352.GA35077@erix.ericsson.se> Message-ID: Not quite, it is really 3 processes: a "master" process which keeps track of the others; an evaluator process which does all the work; and a process which reads the inputs. It is the evaluator process which crashes when there is an error and it is the one which receives messages and is linked. The shell master does not keep track of the evaluator's links so when it dies they all go away. Erlang/OTP 19 [erts-8.0] [source-6dc93c1] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V8.0 (abort with ^G) 1> self(). <0.57.0> 2> process_info(self(), links). {links,[<0.51.0>]} 3> link(whereis(error_logger)). true 4> process_info(self(), links). {links,[<0.51.0>,<0.30.0>]} 5> exit(die_die_die). ** exception exit: die_die_die 6> self(). <0.63.0> 7> process_info(self(), links). {links,[<0.51.0>]} 8> link(whereis(error_logger)). true 9> process_info(self(), links). {links,[<0.51.0>,<0.30.0>]} 10> exit(normal). ** exception exit: normal 11> self(). <0.69.0> 12> process_info(self(), links). {links,[<0.51.0>]} 13> But yes, the shell master keeps track of the process dictionary and the variable bindings and passes them on to the next evaluator process. The evaluator process does nothing special to catch errors and quite happily just dies when there is an error as it knows the master process will handle. It's the "let it crash" philosophy even here. Robert On 12 January 2017 at 13:43, Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > On Thu, Jan 12, 2017 at 09:58:06AM +0000, Roger Lipscombe wrote: > > On 12 January 2017 at 08:41, Raimo Niskanen > > wrote: > > > It is a shell feature. > > > > > > Notice below - an "** exception error:" is different from > > > "*** Shell process terminated! ***". 1/0 is as well as most other > > > exceptions is caught by the evaluator so the process will not exit. > > > > Except that the process *does* exit (note that my shell displays the > > pid in the prompt, but I've added self() calls to be clear): > > > > Erlang/OTP 17 [erts-6.4.1.2] [source-73fc7ca] [64-bit] [smp:12:12] > > [async-threads:10] [hipe] [kernel-poll:false] > > > > Eshell V6.4.1.2 (abort with ^G) > > <0.33.0> 1> self(). > > <0.33.0> > > <0.33.0> 2> put(a, 1). > > undefined > > <0.33.0> 3> get(). > > [{a,1}] > > <0.33.0> 4> 1/0. > > ** exception error: an error occurred when evaluating an arithmetic > expression > > in operator '/'/2 > > called as 1 / 0 > > <0.38.0> 5> self(). > > <0.38.0> > > <0.38.0> 6> get(). > > [{a,1}] > > Well, how about that!? > > The shell does strange things. It is for example divided into two > processes; one evaluator process and one other process that I think for > instance owns links, receives messages, etc. > > It seems the shell emulates what I said i.e that an exception was caught, > but the process receiving messages actually died due to the exception. > > Confusing. I wonder if it is possible to correct...? > > Nevertheless, if a non-emulated process dies due to a runtime error e.g > division by zero, its process dictionary, links and other process stuff > dies with it. > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Fri Jan 13 17:09:54 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 13 Jan 2017 17:09:54 +0100 Subject: [erlang-questions] enif_send rules questions In-Reply-To: References: Message-ID: <612e8bad-f9cb-312d-0c90-f03d9f1ebe2f@ericsson.com> On 01/12/2017 08:21 PM, Jason Orendorff wrote: > Hi all, > > The documentation for enif_send says this: > > env > The environment of the calling process. Must be NULL only if > calling from a created thread. > > I have a few questions about this. > > 1. What is "the environment of the calling process"? Does that mean the > environment that was passed to the NIF? Yes. > 2. What is a "created thread"? Any non-scheduler thread where you don't have "the environment of the calling process". Typically a thread created with enif_thread_create(). > 3. Is it really invalid to do this from a NIF call -- unless I spawn a > separate thread to do it, and then it's OK? > > NIF_ENV msg_env = enif_alloc_env(); > NIF_TERM msg = enif_make_int(env, 1); > enif_send(NULL, &pid, msg_env, msg); > enif_free_env(msg_env); Yes. Documentation contributions to make thing more clear are always welcome. > 4. We suspect it's actually totally safe to pass a process-independent > environment as the first parameter, regardless of whether there's a > "calling process" or we're "calling from a created thread". Is it? > If so, would you accept a patch to document that that's OK? I'm not super excited about such a guarantee, at least not right now. Even if you could conclude that it will work with current implementation, the more guarantees we give, the harder it gets to do future changes and optimizations. Also master branch is a moving target in this area right now as we're in the process of making the dirty scheduler feature more robust for OTP 20. /Sverker, Erlang/OTP From andras.boroska@REDACTED Fri Jan 13 17:24:50 2017 From: andras.boroska@REDACTED (=?UTF-8?Q?Boroska_Andr=C3=A1s?=) Date: Fri, 13 Jan 2017 17:24:50 +0100 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> Message-ID: Hi Tony, The OTP team at Ericsson stopped supporting AIX a while back (~10 years ago). The good news is that there was AIX support in the past and we can build on it. Since then both Erlang, AIX and CPUs evolved. I managed to run Erlang 17 on AIX 7.1 on Power7 and Power8 architectures with minimal sacrifice of features.* 64 bit is supported and worked fine, but naturally required minimal changes in makefiles. So, it is not hopeless. For start, look into the automake files (Erlang uses autoconf). You will need to modify some linker flags. Get prepared to modify some assembly code too. AIX has its own assembler. My strategy was: start with R15, fix things, move to R1602 fix things, move to R1603 fix things, move to R17, fix things. In each step applied fixes from the version before. Erlang 18 brought breaking changes by the new time handling, and I had no time to fix that. I did not try Erlang 19. My port is sadly closed source currently. I wish IBM or related companies sponsored the OTP team to support AIX. Currently it is much easier to run Linux on Power CPU and forget about AIX. Happy hacking, Andras *Supercarriers and kernel poll was not working. On the other hand binding schedulers to cores did work after minimal changes to source. Getting kernel poll working would be a few days effort probably. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.reix@REDACTED Fri Jan 13 18:07:36 2017 From: tony.reix@REDACTED (REIX, Tony) Date: Fri, 13 Jan 2017 17:07:36 +0000 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> Message-ID: Hi Andras, Thanks for your answer ! :) Sad that your source is a closed source. :( About your sentence: "forget about AIX", I'm not so sure, since AIX still provides many features that Linux does not provide yet. About the port, yes I've already started to modify the assembly code. Not too difficult with the help of our expert. I guess that I'll have to add the XCOFF at some time and to manage some more stuff specific to AIX, but up to now it was not so difficult... except that I have NO idea about the amount of work that remains for finalizing the port with ALL required & interesting features (and with good performance). Build is now blocked at: ERLC ../ebin/hipe_rtl.beam and I have no idea why it takes so long and how many steps are still required before the build is complete. I should build Erlang on Linux/x86 in order to see how long it takes. Blocked command is: /opt/freeware/src/packages/BUILD/otp_src_19.2/bin/powerpc-ibm-aix6.1/beam.smp -sbtu -A0 -- -root /opt/freeware/src/packages/BUILD/otp_src_19.2/bootstrap -progname erl -- -home / -- -noshell -noinput -mode minimal -boot start_clean -s erl_compile compile_cmdline -extra -W +debug_info -Werror +inline +warn_unused_import +warn_export_vars -o../ebin hipe_rtl.erl No idea about what it does... Are there some documents describing the main phases of building Erlang ? what are the main features that must be made available ? I see that Erlang does some bootstrap ? Is it described in some document ? About "autoconf", for now, that seems to have worked quite good without modifying this. Unless it has messed up things silently... I'll have a look at linker flags. My "strategy" was to not look at our old port 15B02 . Rather start with 19.2 directly. However, I'll have a look at our old patches then. Anyone interested by Erlang on AIX can contact me. I will share my patches with pleasure. It is OpenSource and, once finalized, it will be made available on BullFreeware web-site . Thanks/Regards, Tony Reix http://bullfreeware.com/toolbox.php Le 13/01/2017 ? 17:24, Boroska Andr?s a ?crit : Hi Tony, The OTP team at Ericsson stopped supporting AIX a while back (~10 years ago). The good news is that there was AIX support in the past and we can build on it. Since then both Erlang, AIX and CPUs evolved. I managed to run Erlang 17 on AIX 7.1 on Power7 and Power8 architectures with minimal sacrifice of features.* 64 bit is supported and worked fine, but naturally required minimal changes in makefiles. So, it is not hopeless. For start, look into the automake files (Erlang uses autoconf). You will need to modify some linker flags. Get prepared to modify some assembly code too. AIX has its own assembler. My strategy was: start with R15, fix things, move to R1602 fix things, move to R1603 fix things, move to R17, fix things. In each step applied fixes from the version before. Erlang 18 brought breaking changes by the new time handling, and I had no time to fix that. I did not try Erlang 19. My port is sadly closed source currently. I wish IBM or related companies sponsored the OTP team to support AIX. Currently it is much easier to run Linux on Power CPU and forget about AIX. Happy hacking, Andras *Supercarriers and kernel poll was not working. On the other hand binding schedulers to cores did work after minimal changes to source. Getting kernel poll working would be a few days effort probably. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Fri Jan 13 19:16:52 2017 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Fri, 13 Jan 2017 10:16:52 -0800 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: On 01/13/2017 01:39 AM, Ingela Andin wrote: > When it comes to security you should be very careful... Agreed. > However we would not want connection_information to return these values > by default. That's sensible. master_secret, server_random, and client_random should be documented options (with a "THIS INFORMATION IS SECURITY SENSITIVE! BE CAREFUL!" warning attached to them) for ssl:connection_information/2. However, what does it mean to start a connection in "debug mode"? The string "debug" neither appears in the ssl module documentation, nor in its source code (OTP 19.2). Isn't limiting access to those parameters to an explicit request (via ssl:connection_information/2) a sufficiently high barrier? Why require one to start a connection in a special mode in addition to making the explicit request for the parameters? > What if someone thinks its a good idea to decrypt the data > outside the TLS connection in the server and send it to an > external logging server in the clear! The server already has the plaintext that was transferred through the TLS tunnel. TLS doesn't protect you from your peer, it protects you from someone in the middle. :) > What if someone decides to transfer the logs in an insecure way from > the server! Sure, but "malicious or poorly-thought-out TLS peers (or system operators)" are -AFAICT- outside the scope of the protection that TLS provides. Given that both OpenSSL and BoringSSL provide functions to extract client/server random and master secret, it seems that there is a legitimate (if niche) reason for exposing this information to clients. I mean, the Erlang ssl module allows you to turn off BEAST mitigation and the block cypher padding check. These options are arguably more dangerous than what's being discussed (as they weaken TLS's MITM protections (leaving you open to POODLE and BEAST)) but they are _very_ useful in certain niche situations. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ingela.andin@REDACTED Fri Jan 13 19:57:15 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 13 Jan 2017 19:57:15 +0100 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: Hi! 2017-01-13 19:16 GMT+01:00 Kenneth Lakin : > On 01/13/2017 01:39 AM, Ingela Andin wrote: > > When it comes to security you should be very careful... > > Agreed. > > > However we would not want connection_information to return these values > > by default. > > That's sensible. master_secret, server_random, and client_random should > be documented options (with a "THIS INFORMATION IS SECURITY SENSITIVE! > BE CAREFUL!" warning attached to them) for ssl:connection_information/2. > > However, what does it mean to start a connection in "debug mode"? The > string "debug" neither appears in the ssl module documentation, nor in > its source code (OTP 19.2). Isn't limiting access to those parameters to > an explicit request (via ssl:connection_information/2) a sufficiently > high barrier? Why require one to start a connection in a special mode in > addition to making the explicit request for the parameters? > "debug mode" would be a new option (maybe called something else) that when set connection_information would be allowed to return master_secret, server_random, and client_random. This to ensure that it is the intent of the connection starter that this behaviour should be allowed. > > What if someone thinks its a good idea to decrypt the data > > outside the TLS connection in the server and send it to an > > external logging server in the clear! > > The server already has the plaintext that was transferred through the > TLS tunnel. TLS doesn't protect you from your peer, it protects you from > someone in the middle. :) > > Well yes the peer can choose to handle the plain-text uncarfully and TLS can do nothing about it. I think there is a reason for the joke (a)lawful interception ;) The point here is that if it seems to standard (normal API) people might use it out of convinance for testing/debuging or not understanding better and end up compromising security. > > What if someone decides to transfer the logs in an insecure way from > > the server! > > Sure, but "malicious or poorly-thought-out TLS peers (or system > operators)" are -AFAICT- outside the scope of the protection that TLS > provides. Given that both OpenSSL and BoringSSL provide functions to > extract client/server random and master secret, it seems that there is a > legitimate (if niche) reason for exposing this information to clients. > > I mean, the Erlang ssl module allows you to turn off BEAST mitigation > and the block cypher padding check. These options are arguably more > dangerous than what's being discussed (as they weaken TLS's MITM > protections (leaving you open to POODLE and BEAST)) but they are _very_ > useful in certain niche situations. > > Yes interop vs security can be a tradeoff. All these needs the user to make an active choise. Regards Ingela Erlang/OTP team - Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Fri Jan 13 21:27:29 2017 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Fri, 13 Jan 2017 12:27:29 -0800 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: References: Message-ID: <92895e0e-91df-2239-7eb7-bd068fab7759@gmail.com> On 01/13/2017 10:57 AM, Ingela Andin wrote: > Yes interop vs security can be a tradeoff. All these needs the user to make > an active choise. Notably, that active choice _doesn't_ include forcing the programmer to also start the connection in "interop mode". Marking those ssl_options with Big Red Dire Warnings was (correctly) deemed quite enough notice. :) Heck, verify_fun is documented as normal, non-hazardous (that is, it lacks a Big Red Warning box) API, but can be misused to (accidentally) seriously compromise one's connection security. verify_fun does _not_ depend on an "enable_hazardous_options" connection flag. > This to ensure that it is the intent of the connection starter > that this behaviour should be allowed. Given that the caller can bypass this check by pulling the PID out of the SSLSocket, calling sys:get_state/1 and extracting the security_parameters, this check seems to be more of an annoyance than anything else. For instance, how would the author of the (hypothetical) TLS connection pool library I'm using know that I _never_ will have a legitimate need to extract the client_random from a connection it establishes for me? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From steven.charles.davis@REDACTED Fri Jan 13 23:26:02 2017 From: steven.charles.davis@REDACTED (Steve Davis) Date: Fri, 13 Jan 2017 16:26:02 -0600 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: <823461E8-359A-4A2D-BC22-0B352919CB72@gmail.com> > On Jan 12, 2017, at 1:56 PM, Jesper Louis Andersen wrote: > Is Erlang bad at string processing? Yes. Every language is bad at "string processing?. IMHO a better question is: Is Erlang bad at text processing? No. You just need to realize that there?s *no such thing* as a string in erlang, and keep ill-specified text binaries as binaries. Then you can use the magic of (cautious and limited) regex for tokenization and binary syntax (which is ***truly wonderful***) for parsing. So do I have problems with text in Erlang? No, it?s far better for that than any other language I?ve used (and I?ve used quite a few). Strings suck as a way to think about text. That?s the sum of it. My 2c on all this. :-) From michal@REDACTED Fri Jan 13 23:34:55 2017 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Fri, 13 Jan 2017 23:34:55 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: I fully agree there are no languages that deal with strings perfectly. That said there are those that are better at it and those that aren't so good. A language, where I need to look for a library to upcase or downcase my own name, fits into the second group in my book. Micha?. On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen , wrote: > Richard is indeed right, depending on what your definition of "String" is. > > If a "String" is "An array of characters from some alphabet", then you need to take into account Strings are Unicode codepoints in practice. This is also the most precise definition from a technical point of view. > > When I wrote my post, I was--probably incorrectly--assuming the older notion of a "String" where the representation is either ASCII or something like ISO-8859-15. In this case, a string coincides with a stream of bytes. > > Data needs parsing. A lot of data comes in as some kind of stringy representation: UTF-8, byte array (binary), and so on. > > And of course, that isn't the whole story, since there are examples of input which are not string-like in their forms. > > > > On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe wrote: > > > > > > > > > On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: > > > > Strings are really just streams of bytes. > > > > > > That was true a long time ago.? Maybe. > > > But it isn't anywhere near accurate as a description > > > of Unicode: > > > ? - Unicode is made of 21-bit code points, not bytes. > > > ? - Most possible code points are not defined. > > > ? - Some of those that are defined are defined as > > > ? ? "it is illegal to use this". > > > ? - Unicode sequences have *structure*; it is simply > > > ? ? not the case that every sequence of allowable > > > ? ? Unicode code points is a legal Unicode string. > > > ? - As a special case of that, if s is a non-empty > > > ? ? valid Unicode string, it is not true that every > > > ? ? substring of s is a valid Unicode string. > > > > > > In case you were thinking of UTF-8, not all byte > > > sequences are valid UTF-8. > > > > > > Byte streams are as important as you say, but it's > > > really hard to see the software for a radar or a > > > radio telescope as processing strings... > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.orendorff@REDACTED Fri Jan 13 23:53:07 2017 From: jason.orendorff@REDACTED (Jason Orendorff) Date: Fri, 13 Jan 2017 16:53:07 -0600 Subject: [erlang-questions] enif_send rules questions In-Reply-To: <612e8bad-f9cb-312d-0c90-f03d9f1ebe2f@ericsson.com> References: <612e8bad-f9cb-312d-0c90-f03d9f1ebe2f@ericsson.com> Message-ID: On Fri, Jan 13, 2017 at 10:09 AM, Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > On 01/12/2017 08:21 PM, Jason Orendorff wrote: > >> 4. We suspect it's actually totally safe to pass a process-independent >> environment as the first parameter, regardless of whether there's a >> "calling process" or we're "calling from a created thread". Is it? >> If so, would you accept a patch to document that that's OK? >> > I'm not super excited about such a guarantee, at least not right now. > Thank you for the quick, thoughtful response. I appreciate your time. Is there a way to enforce this rule for our users that we've missed? When our `send` method is called, I think we have to figure out if there's a current calling process, and if so, its environment. I don't see a way to do it without storing that information in TLS ahead of time -- at a cost to every NIF call. We could make code that runs on created threads statically different from code that runs in the Erlang scheduler's threads, e.g. by making `NifCallEnv`, `AllocatedEnvInNifThread`, and `AllocatedEnvInCreatedThread` three different types, of which only 2 have `.send()` methods. But this seems horrible. We have considered several possible approaches, all about as bad as that. Thanks, -j -------------- next part -------------- An HTML attachment was scrubbed... URL: From himars@REDACTED Sat Jan 14 02:39:39 2017 From: himars@REDACTED (Jack Tang) Date: Sat, 14 Jan 2017 09:39:39 +0800 Subject: [erlang-questions] How can I release beam process memory? Message-ID: Hello list, I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command. ``` KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers KiB Swap: 0 total, 0 used, 0 free. 18684864 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp ``` I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow: ``` lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children(some_thing_sup)). ``` and erlang:memory() decreases from 32G to 23G. ``` [{total,22982011544}, {processes,12884182336}, {processes_used,12884170336}, {system,10097829208}, {atom,13828705}, {atom_used,13796692}, {binary,170530288}, {code,16450626}, {ets,9637717576}] ``` However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks BR -Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sat Jan 14 14:00:35 2017 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 14 Jan 2017 21:00:35 +0800 Subject: [erlang-questions] enif_consume_timeslice performances In-Reply-To: References: Message-ID: What do you call a "performance"? Do you measure throughput on thousands of simultaneous connections? Or you measure throughput on 2-3 simultaneous connections? How do you simulate network drops? On Thu, Jan 12, 2017 at 11:30 PM, Caragea Silviu wrote: > Hello, > > I'm working on a TLS/DTLS library in top of BoringSSL/OpenSSL and gen_tcp. > My goal is to have a interface compatible as much as I can with erlang ssl > module. > > Project is located here: https://github.com/silviucpp/erltls and overall > so far I have better performances results than ssl module. around 33 % on > OSX and 18 % on ubuntu when using ssl as client for sending the requests > (when using erltls as client boost is even more for example on mac around > 60 %) > > To measure the performances I'm using : https://github.com/silviucpp/ > tls_bench > > During testing I observed that when using enif_consume_timeslice I have a > drop off in performances for around 15 %. It's calling this method too > heavy ? or the processes are suspended/resumed to many times and this > affects the results? > > Silviu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donpedrothird@REDACTED Sat Jan 14 15:18:40 2017 From: donpedrothird@REDACTED (John Doe) Date: Sat, 14 Jan 2017 17:18:40 +0300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: Indeed, unicode upercase/lowercsase is one of the most essential features of string which don't exist yet in erlang stdlib. I'm aware about problems with some letters and scripts, such as german SS or turkish I, but still having upper/lower in stdlib is the must, IMO. The problem is that uppercase/lowercase would require support of unicode normalization. 2017-01-14 1:34 GMT+03:00 Micha? Muska?a : > I fully agree there are no languages that deal with strings perfectly. > That said there are those that are better at it and those that aren't so > good. A language, where I need to look for a library to upcase or downcase > my own name, fits into the second group in my book. > > Micha?. > > On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED>, wrote: > > Richard is indeed right, depending on what your definition of "String" is. > > If a "String" is "An array of characters from some alphabet", then you > need to take into account Strings are Unicode codepoints in practice. This > is also the most precise definition from a technical point of view. > > When I wrote my post, I was--probably incorrectly--assuming the older > notion of a "String" where the representation is either ASCII or something > like ISO-8859-15. In this case, a string coincides with a stream of bytes. > > Data needs parsing. A lot of data comes in as some kind of stringy > representation: UTF-8, byte array (binary), and so on. > > And of course, that isn't the whole story, since there are examples of > input which are not string-like in their forms. > > > On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe > wrote: > >> >> >> On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: >> > Strings are really just streams of bytes. >> >> That was true a long time ago. Maybe. >> But it isn't anywhere near accurate as a description >> of Unicode: >> - Unicode is made of 21-bit code points, not bytes. >> - Most possible code points are not defined. >> - Some of those that are defined are defined as >> "it is illegal to use this". >> - Unicode sequences have *structure*; it is simply >> not the case that every sequence of allowable >> Unicode code points is a legal Unicode string. >> - As a special case of that, if s is a non-empty >> valid Unicode string, it is not true that every >> substring of s is a valid Unicode string. >> >> In case you were thinking of UTF-8, not all byte >> sequences are valid UTF-8. >> >> Byte streams are as important as you say, but it's >> really hard to see the software for a radar or a >> radio telescope as processing strings... >> >> _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sat Jan 14 15:56:21 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Sat, 14 Jan 2017 15:56:21 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: We need support for locales before we can do proper operations on text. Just Unicode isn't enough. On 01/14/2017 03:18 PM, John Doe wrote: > Indeed, unicode upercase/lowercsase is one of the most essential > features of string which don't exist yet in erlang stdlib. I'm aware > about problems with some letters and scripts, such as german SS or > turkish I, but still having upper/lower in stdlib is the must, IMO. The > problem is that uppercase/lowercase would require support of unicode > normalization. > > 2017-01-14 1:34 GMT+03:00 Micha? Muska?a >: > > I fully agree there are no languages that deal with strings > perfectly. That said there are those that are better at it and those > that aren't so good. A language, where I need to look for a library > to upcase or downcase my own name, fits into the second group in my > book. > > > Micha?. > > On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen > >, wrote: >> Richard is indeed right, depending on what your definition of >> "String" is. >> >> If a "String" is "An array of characters from some alphabet", then >> you need to take into account Strings are Unicode codepoints in >> practice. This is also the most precise definition from a >> technical point of view. >> >> When I wrote my post, I was--probably incorrectly--assuming the >> older notion of a "String" where the representation is either >> ASCII or something like ISO-8859-15. In this case, a string >> coincides with a stream of bytes. >> >> Data needs parsing. A lot of data comes in as some kind of stringy >> representation: UTF-8, byte array (binary), and so on. >> >> And of course, that isn't the whole story, since there are >> examples of input which are not string-like in their forms. >> >> >> On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe >> > wrote: >> >> >> >> On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: >> > Strings are really just streams of bytes. >> >> That was true a long time ago. Maybe. >> But it isn't anywhere near accurate as a description >> of Unicode: >> - Unicode is made of 21-bit code points, not bytes. >> - Most possible code points are not defined. >> - Some of those that are defined are defined as >> "it is illegal to use this". >> - Unicode sequences have *structure*; it is simply >> not the case that every sequence of allowable >> Unicode code points is a legal Unicode string. >> - As a special case of that, if s is a non-empty >> valid Unicode string, it is not true that every >> substring of s is a valid Unicode string. >> >> In case you were thinking of UTF-8, not all byte >> sequences are valid UTF-8. >> >> Byte streams are as important as you say, but it's >> really hard to see the software for a radar or a >> radio telescope as processing strings... >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From bchesneau@REDACTED Sat Jan 14 16:29:59 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 14 Jan 2017 15:29:59 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS: https://github.com/erlang-unicode/i18n - beno?t On Sat, Jan 14, 2017 at 3:56 PM Lo?c Hoguin wrote: > We need support for locales before we can do proper operations on text. > Just Unicode isn't enough. > > On 01/14/2017 03:18 PM, John Doe wrote: > > Indeed, unicode upercase/lowercsase is one of the most essential > > features of string which don't exist yet in erlang stdlib. I'm aware > > about problems with some letters and scripts, such as german SS or > > turkish I, but still having upper/lower in stdlib is the must, IMO. The > > problem is that uppercase/lowercase would require support of unicode > > normalization. > > > > 2017-01-14 1:34 GMT+03:00 Micha? Muska?a > >: > > > > I fully agree there are no languages that deal with strings > > perfectly. That said there are those that are better at it and those > > that aren't so good. A language, where I need to look for a library > > to upcase or downcase my own name, fits into the second group in my > > book. > > > > > > Micha?. > > > > On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen > > > >, wrote: > >> Richard is indeed right, depending on what your definition of > >> "String" is. > >> > >> If a "String" is "An array of characters from some alphabet", then > >> you need to take into account Strings are Unicode codepoints in > >> practice. This is also the most precise definition from a > >> technical point of view. > >> > >> When I wrote my post, I was--probably incorrectly--assuming the > >> older notion of a "String" where the representation is either > >> ASCII or something like ISO-8859-15. In this case, a string > >> coincides with a stream of bytes. > >> > >> Data needs parsing. A lot of data comes in as some kind of stringy > >> representation: UTF-8, byte array (binary), and so on. > >> > >> And of course, that isn't the whole story, since there are > >> examples of input which are not string-like in their forms. > >> > >> > >> On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe > >> > wrote: > >> > >> > >> > >> On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: > >> > Strings are really just streams of bytes. > >> > >> That was true a long time ago. Maybe. > >> But it isn't anywhere near accurate as a description > >> of Unicode: > >> - Unicode is made of 21-bit code points, not bytes. > >> - Most possible code points are not defined. > >> - Some of those that are defined are defined as > >> "it is illegal to use this". > >> - Unicode sequences have *structure*; it is simply > >> not the case that every sequence of allowable > >> Unicode code points is a legal Unicode string. > >> - As a special case of that, if s is a non-empty > >> valid Unicode string, it is not true that every > >> substring of s is a valid Unicode string. > >> > >> In case you were thinking of UTF-8, not all byte > >> sequences are valid UTF-8. > >> > >> Byte streams are as important as you say, but it's > >> really hard to see the software for a radar or a > >> radio telescope as processing strings... > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > >> > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Sat Jan 14 16:53:01 2017 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Sat, 14 Jan 2017 16:53:01 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> , Message-ID: Could the Unicode support in elixir serve as a starting point? https://hexdocs.pm/elixir/1.3.3/String.html#content String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: "For example, the codepoint ??? is two bytes: iex> byte_size("?") 2" Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... Cheers, Oliver ? ? Gesendet:?Freitag, 13. Januar 2017 um 23:34 Uhr Von:?"Micha? Muska?a" An:?"Richard A. O'Keefe" , "Steve Davis" , g@REDACTED, "Jesper Louis Andersen" Cc:?"Erlang Questions" Betreff:?Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead I fully agree there are no languages that deal with strings perfectly. That said there are those that are better at it and those that aren't so good. A language, where I need to look for a library to upcase or downcase my own name, fits into the second group in my book. Micha?. On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen , wrote: Richard is indeed right, depending on what your definition of "String" is. ?If a "String" is "An array of characters from some alphabet", then you need to take into account Strings are Unicode codepoints in practice. This is also the most precise definition from a technical point of view. ?When I wrote my post, I was--probably incorrectly--assuming the older notion of a "String" where the representation is either ASCII or something like ISO-8859-15. In this case, a string coincides with a stream of bytes. ?Data needs parsing. A lot of data comes in as some kind of stringy representation: UTF-8, byte array (binary), and so on. ?And of course, that isn't the whole story, since there are examples of input which are not string-like in their forms. ?? On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe wrote: On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: > Strings are really just streams of bytes. That was true a long time ago.? Maybe. But it isn't anywhere near accurate as a description of Unicode: ? - Unicode is made of 21-bit code points, not bytes. ? - Most possible code points are not defined. ? - Some of those that are defined are defined as ? ? "it is illegal to use this". ? - Unicode sequences have *structure*; it is simply ? ? not the case that every sequence of allowable ? ? Unicode code points is a legal Unicode string. ? - As a special case of that, if s is a non-empty ? ? valid Unicode string, it is not true that every ? ? substring of s is a valid Unicode string. In case you were thinking of UTF-8, not all byte sequences are valid UTF-8. Byte streams are as important as you say, but it's really hard to see the software for a radar or a radio telescope as processing strings... ?_______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] From ilya.khaprov@REDACTED Sat Jan 14 17:00:52 2017 From: ilya.khaprov@REDACTED (Ilya Khaprov) Date: Sat, 14 Jan 2017 16:00:52 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> , , Message-ID: >> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? Why Elixir implements Unicode in Elixir? You have to rewrite it anyway. Ilya From: Oliver Korpilla Sent: Saturday, January 14, 2017 06:53 PM To: Micha? Muska?a Cc: Erlang Questions Subject: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead Could the Unicode support in elixir serve as a starting point? https://hexdocs.pm/elixir/1.3.3/String.html#content String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: "For example, the codepoint ??? is two bytes: iex> byte_size("?") 2" Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... Cheers, Oliver Gesendet: Freitag, 13. Januar 2017 um 23:34 Uhr Von: "Micha? Muska?a" An: "Richard A. O'Keefe" , "Steve Davis" , g@REDACTED, "Jesper Louis Andersen" Cc: "Erlang Questions" Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead I fully agree there are no languages that deal with strings perfectly. That said there are those that are better at it and those that aren't so good. A language, where I need to look for a library to upcase or downcase my own name, fits into the second group in my book. Micha?. On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen , wrote: Richard is indeed right, depending on what your definition of "String" is. If a "String" is "An array of characters from some alphabet", then you need to take into account Strings are Unicode codepoints in practice. This is also the most precise definition from a technical point of view. When I wrote my post, I was--probably incorrectly--assuming the older notion of a "String" where the representation is either ASCII or something like ISO-8859-15. In this case, a string coincides with a stream of bytes. Data needs parsing. A lot of data comes in as some kind of stringy representation: UTF-8, byte array (binary), and so on. And of course, that isn't the whole story, since there are examples of input which are not string-like in their forms. On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe wrote: On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: > Strings are really just streams of bytes. That was true a long time ago. Maybe. But it isn't anywhere near accurate as a description of Unicode: - Unicode is made of 21-bit code points, not bytes. - Most possible code points are not defined. - Some of those that are defined are defined as "it is illegal to use this". - Unicode sequences have *structure*; it is simply not the case that every sequence of allowable Unicode code points is a legal Unicode string. - As a special case of that, if s is a non-empty valid Unicode string, it is not true that every substring of s is a valid Unicode string. In case you were thinking of UTF-8, not all byte sequences are valid UTF-8. Byte streams are as important as you say, but it's really hard to see the software for a radar or a radio telescope as processing strings... _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Jan 14 17:06:15 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 14 Jan 2017 16:06:15 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: On Sat, Jan 14, 2017 at 4:53 PM Oliver Korpilla wrote: > Could the Unicode support in elixir serve as a starting point? > > https://hexdocs.pm/elixir/1.3.3/String.html#content > > String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot > of effort seems have gone in scenarios like this: > > "For example, the codepoint ??? is two bytes: > > iex> byte_size("?") > 2" > > Given that both Erlang and elixir are implemented on top of BEAM, the > wheel might not need reinventing? I know engineers and programmers love > inventing stuff, and this discussion seems to point in that direction, > but... > > Cheers, > Oliver > > If I remember correctly the unicode support of Elixir is written in elixir and data come from the unicode/icu projects. data resources (codepoints and so on ) are compiled as beam. (I do the dame in my idna lib). The work may be simpler in using/wrting a nif over the well supported ICU lib thoug. I'm curious about the reasonning that conducted to the current implementation in elixir. - benoit > > > Gesendet: Freitag, 13. Januar 2017 um 23:34 Uhr > Von: "Micha? Muska?a" > An: "Richard A. O'Keefe" , "Steve Davis" < > steven.charles.davis@REDACTED>, g@REDACTED, "Jesper Louis Andersen" < > jesper.louis.andersen@REDACTED> > Cc: "Erlang Questions" > Betreff: Re: [erlang-questions] Erlang basic doubts about String, message > passing and context switching overhead > > I fully agree there are no languages that deal with strings perfectly. > That said there are those that are better at it and those that aren't so > good. A language, where I need to look for a library to upcase or downcase > my own name, fits into the second group in my book. > > Micha?. > On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED>, wrote: > > Richard is indeed right, depending on what your definition of "String" is. > If a "String" is "An array of characters from some alphabet", then you > need to take into account Strings are Unicode codepoints in practice. This > is also the most precise definition from a technical point of view. > When I wrote my post, I was--probably incorrectly--assuming the older > notion of a "String" where the representation is either ASCII or something > like ISO-8859-15. In this case, a string coincides with a stream of bytes. > Data needs parsing. A lot of data comes in as some kind of stringy > representation: UTF-8, byte array (binary), and so on. > And of course, that isn't the whole story, since there are examples of > input which are not string-like in their forms. > > > On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe [mailto:ok@REDACTED]> wrote: > > On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: > > Strings are really just streams of bytes. > > That was true a long time ago. Maybe. > But it isn't anywhere near accurate as a description > of Unicode: > - Unicode is made of 21-bit code points, not bytes. > - Most possible code points are not defined. > - Some of those that are defined are defined as > "it is illegal to use this". > - Unicode sequences have *structure*; it is simply > not the case that every sequence of allowable > Unicode code points is a legal Unicode string. > - As a special case of that, if s is a non-empty > valid Unicode string, it is not true that every > substring of s is a valid Unicode string. > > In case you were thinking of UTF-8, not all byte > sequences are valid UTF-8. > > Byte streams are as important as you say, but it's > really hard to see the software for a radar or a > radio telescope as processing strings... > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________ > erlang-questions mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donpedrothird@REDACTED Sat Jan 14 17:14:58 2017 From: donpedrothird@REDACTED (John Doe) Date: Sat, 14 Jan 2017 19:14:58 +0300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: Nif would make erlang distribution way less portable than it is now. At the moment erlang compiled on with static openss can be moven just by copying between most modern linux distros of the same bits. ICU is a C++ lib, anything with linkage against libstdc++ is not portable, new versions of GCC break compatibility of libstdc++ quite often. 2017-01-14 19:06 GMT+03:00 Benoit Chesneau : > > > On Sat, Jan 14, 2017 at 4:53 PM Oliver Korpilla > wrote: > >> Could the Unicode support in elixir serve as a starting point? >> >> https://hexdocs.pm/elixir/1.3.3/String.html#content >> >> String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot >> of effort seems have gone in scenarios like this: >> >> "For example, the codepoint ??? is two bytes: >> >> iex> byte_size("?") >> 2" >> >> Given that both Erlang and elixir are implemented on top of BEAM, the >> wheel might not need reinventing? I know engineers and programmers love >> inventing stuff, and this discussion seems to point in that direction, >> but... >> >> Cheers, >> Oliver >> >> > > If I remember correctly the unicode support of Elixir is written in elixir > and data come from the unicode/icu projects. data resources (codepoints and > so on ) are compiled as beam. (I do the dame in my idna lib). > > The work may be simpler in using/wrting a nif over the well supported ICU > lib thoug. I'm curious about the reasonning that conducted to the current > implementation in elixir. > > - benoit > > > >> >> >> Gesendet: Freitag, 13. Januar 2017 um 23:34 Uhr >> Von: "Micha? Muska?a" >> An: "Richard A. O'Keefe" , "Steve Davis" < >> steven.charles.davis@REDACTED>, g@REDACTED, "Jesper Louis Andersen" < >> jesper.louis.andersen@REDACTED> >> Cc: "Erlang Questions" >> Betreff: Re: [erlang-questions] Erlang basic doubts about String, message >> passing and context switching overhead >> >> I fully agree there are no languages that deal with strings perfectly. >> That said there are those that are better at it and those that aren't so >> good. A language, where I need to look for a library to upcase or downcase >> my own name, fits into the second group in my book. >> >> Micha?. >> On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen < >> jesper.louis.andersen@REDACTED>, wrote: >> >> Richard is indeed right, depending on what your definition of "String" is. >> If a "String" is "An array of characters from some alphabet", then you >> need to take into account Strings are Unicode codepoints in practice. This >> is also the most precise definition from a technical point of view. >> When I wrote my post, I was--probably incorrectly--assuming the older >> notion of a "String" where the representation is either ASCII or something >> like ISO-8859-15. In this case, a string coincides with a stream of bytes. >> Data needs parsing. A lot of data comes in as some kind of stringy >> representation: UTF-8, byte array (binary), and so on. >> And of course, that isn't the whole story, since there are examples of >> input which are not string-like in their forms. >> >> >> On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe > [mailto:ok@REDACTED]> wrote: >> >> On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: >> > Strings are really just streams of bytes. >> >> That was true a long time ago. Maybe. >> But it isn't anywhere near accurate as a description >> of Unicode: >> - Unicode is made of 21-bit code points, not bytes. >> - Most possible code points are not defined. >> - Some of those that are defined are defined as >> "it is illegal to use this". >> - Unicode sequences have *structure*; it is simply >> not the case that every sequence of allowable >> Unicode code points is a legal Unicode string. >> - As a special case of that, if s is a non-empty >> valid Unicode string, it is not true that every >> substring of s is a valid Unicode string. >> >> In case you were thinking of UTF-8, not all byte >> sequences are valid UTF-8. >> >> Byte streams are as important as you say, but it's >> really hard to see the software for a radar or a >> radio telescope as processing strings... >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions_____ >> __________________________________________ erlang-questions mailing list >> erlang-questions@REDACTED http://erlang.org/mailman/ >> listinfo/erlang-questions[http://erlang.org/mailman/ >> listinfo/erlang-questions] >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donpedrothird@REDACTED Sat Jan 14 17:22:56 2017 From: donpedrothird@REDACTED (John Doe) Date: Sat, 14 Jan 2017 19:22:56 +0300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: Also as far as I remember icu works with utf16, so any call to icu requires encoding binary, which is usually usually utf8, to utf16 and then encoding it back to utf8 binary. 2017-01-14 19:06 GMT+03:00 Benoit Chesneau : > > > On Sat, Jan 14, 2017 at 4:53 PM Oliver Korpilla > wrote: > >> Could the Unicode support in elixir serve as a starting point? >> >> https://hexdocs.pm/elixir/1.3.3/String.html#content >> >> String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot >> of effort seems have gone in scenarios like this: >> >> "For example, the codepoint ??? is two bytes: >> >> iex> byte_size("?") >> 2" >> >> Given that both Erlang and elixir are implemented on top of BEAM, the >> wheel might not need reinventing? I know engineers and programmers love >> inventing stuff, and this discussion seems to point in that direction, >> but... >> >> Cheers, >> Oliver >> >> > > If I remember correctly the unicode support of Elixir is written in elixir > and data come from the unicode/icu projects. data resources (codepoints and > so on ) are compiled as beam. (I do the dame in my idna lib). > > The work may be simpler in using/wrting a nif over the well supported ICU > lib thoug. I'm curious about the reasonning that conducted to the current > implementation in elixir. > > - benoit > > > >> >> >> Gesendet: Freitag, 13. Januar 2017 um 23:34 Uhr >> Von: "Micha? Muska?a" >> An: "Richard A. O'Keefe" , "Steve Davis" < >> steven.charles.davis@REDACTED>, g@REDACTED, "Jesper Louis Andersen" < >> jesper.louis.andersen@REDACTED> >> Cc: "Erlang Questions" >> Betreff: Re: [erlang-questions] Erlang basic doubts about String, message >> passing and context switching overhead >> >> I fully agree there are no languages that deal with strings perfectly. >> That said there are those that are better at it and those that aren't so >> good. A language, where I need to look for a library to upcase or downcase >> my own name, fits into the second group in my book. >> >> Micha?. >> On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen < >> jesper.louis.andersen@REDACTED>, wrote: >> >> Richard is indeed right, depending on what your definition of "String" is. >> If a "String" is "An array of characters from some alphabet", then you >> need to take into account Strings are Unicode codepoints in practice. This >> is also the most precise definition from a technical point of view. >> When I wrote my post, I was--probably incorrectly--assuming the older >> notion of a "String" where the representation is either ASCII or something >> like ISO-8859-15. In this case, a string coincides with a stream of bytes. >> Data needs parsing. A lot of data comes in as some kind of stringy >> representation: UTF-8, byte array (binary), and so on. >> And of course, that isn't the whole story, since there are examples of >> input which are not string-like in their forms. >> >> >> On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe > [mailto:ok@REDACTED]> wrote: >> >> On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: >> > Strings are really just streams of bytes. >> >> That was true a long time ago. Maybe. >> But it isn't anywhere near accurate as a description >> of Unicode: >> - Unicode is made of 21-bit code points, not bytes. >> - Most possible code points are not defined. >> - Some of those that are defined are defined as >> "it is illegal to use this". >> - Unicode sequences have *structure*; it is simply >> not the case that every sequence of allowable >> Unicode code points is a legal Unicode string. >> - As a special case of that, if s is a non-empty >> valid Unicode string, it is not true that every >> substring of s is a valid Unicode string. >> >> In case you were thinking of UTF-8, not all byte >> sequences are valid UTF-8. >> >> Byte streams are as important as you say, but it's >> really hard to see the software for a radar or a >> radio telescope as processing strings... >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions_____ >> __________________________________________ erlang-questions mailing list >> erlang-questions@REDACTED http://erlang.org/mailman/ >> listinfo/erlang-questions[http://erlang.org/mailman/ >> listinfo/erlang-questions] >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Jan 14 17:32:33 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 14 Jan 2017 16:32:33 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: On Sat, Jan 14, 2017 at 5:23 PM John Doe wrote: > Also as far as I remember icu works with utf16, so any call to icu > requires encoding binary, which is usually usually utf8, to utf16 and then > encoding it back to utf8 binary. > > ICU is far more portable than even Erlang is, working on very different platforms and can now be compiled using clang and not only g++. It is handling Unicode 9 and should work with utf8 also but not sure about the details. Anyway not saying it is better or not hence why I was asking for the reasoning of the Elixir people to come to their solution ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Sat Jan 14 17:36:22 2017 From: g@REDACTED (Garrett Smith) Date: Sat, 14 Jan 2017 10:36:22 -0600 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: On Fri, Jan 13, 2017 at 4:34 PM, Micha? Muska?a wrote: > I fully agree there are no languages that deal with strings perfectly. That > said there are those that are better at it and those that aren't so good. A > language, where I need to look for a library to upcase or downcase my own > name, fits into the second group in my book. If you're talking about this: > "hello".upcase() "HELLO" vs: > string:to_upper("hello"). "HELLO" I would be tempted to rephrase "Erlang is not good for ..." with "Erlang is not what I am used to for ..." Some languages invest tremendous effort in programmer convenience and fit and finish. I think this is terrific! It's one of the major appeals of Elixir vis-a-vis Erlang and has inspired a huge influx of creativity and contributions within that ecosystem. However, when it comes to the merits of a language (and it's libraries, runtime environments, etc.) there are trade offs *everywhere* and some of these conveniences come at a high cost. I don't think "good" and "bad" are nearly specific enough to help inform our decisions about language adoption. Now the following is *my very personal opinion* and I'm not grinding any ax here, extremely happy to live and let live, but this: I don't particularly find writing function(Arg) (as opposed to Arg.function) hard, at all - and I *certainly* don't want to pay *any* price in terms of added complexity or performance degradation for object oriented ish semantics or features. That's me though. I know a lot of people really like their language features and thank goodness we have options! From donpedrothird@REDACTED Sat Jan 14 18:08:20 2017 From: donpedrothird@REDACTED (John Doe) Date: Sat, 14 Jan 2017 20:08:20 +0300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: >string:to_upper("hello"). This function works for ASCII only. That's the problem. MICHA? = string:to_upper("Micha?"). 2017-01-14 19:36 GMT+03:00 Garrett Smith : > On Fri, Jan 13, 2017 at 4:34 PM, Micha? Muska?a wrote: > > I fully agree there are no languages that deal with strings perfectly. > That > > said there are those that are better at it and those that aren't so > good. A > > language, where I need to look for a library to upcase or downcase my own > > name, fits into the second group in my book. > > If you're talking about this: > > > "hello".upcase() > "HELLO" > > vs: > > > string:to_upper("hello"). > "HELLO" > > I would be tempted to rephrase "Erlang is not good for ..." with > "Erlang is not what I am used to for ..." > > Some languages invest tremendous effort in programmer convenience and > fit and finish. I think this is terrific! It's one of the major > appeals of Elixir vis-a-vis Erlang and has inspired a huge influx of > creativity and contributions within that ecosystem. > > However, when it comes to the merits of a language (and it's > libraries, runtime environments, etc.) there are trade offs > *everywhere* and some of these conveniences come at a high cost. I > don't think "good" and "bad" are nearly specific enough to help inform > our decisions about language adoption. > > Now the following is *my very personal opinion* and I'm not grinding > any ax here, extremely happy to live and let live, but this: I don't > particularly find writing function(Arg) (as opposed to Arg.function) > hard, at all - and I *certainly* don't want to pay *any* price in > terms of added complexity or performance degradation for object > oriented ish semantics or features. That's me though. I know a lot of > people really like their language features and thank goodness we have > options! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Sat Jan 14 18:25:35 2017 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Sat, 14 Jan 2017 18:25:35 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> , Message-ID: Hello, to be fair, the Polish special "?" is easy to miss for everybody not versed with the language when doing a visual scan. ;) Which is kind of the point of lots of Unicode code points. But as John Doe pointed out, this is indeed what this is about. It actually Unicode-aware in its conversions which I think is also shown in the examples. Cheers, Oliver? Gesendet:?Samstag, 14. Januar 2017 um 18:08 Uhr Von:?"John Doe" An:?Kein Empf?nger Cc:?"Erlang Questions" Betreff:?Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead >string:to_upper("hello"). ? This function works for ASCII only. That's the problem.?MICHA? =?string:to_upper("Micha?"). ? 2017-01-14 19:36 GMT+03:00 Garrett Smith :On Fri, Jan 13, 2017 at 4:34 PM, Micha? Muska?a wrote: > I fully agree there are no languages that deal with strings perfectly. That > said there are those that are better at it and those that aren't so good. A > language, where I need to look for a library to upcase or downcase my own > name, fits into the second group in my book. If you're talking about this: > "hello".upcase() "HELLO" vs: > string:to_upper("hello"). "HELLO" I would be tempted to rephrase "Erlang is not good for ..." with "Erlang is not what I am used to for ..." Some languages invest tremendous effort in programmer convenience and fit and finish. I think this is terrific! It's one of the major appeals of Elixir vis-a-vis Erlang and has inspired a huge influx of creativity and contributions within that ecosystem. However, when it comes to the merits of a language (and it's libraries, runtime environments, etc.) there are trade offs *everywhere* and some of these conveniences come at a high cost. I don't think "good" and "bad" are nearly specific enough to help inform our decisions about language adoption. Now the following is *my very personal opinion* and I'm not grinding any ax here, extremely happy to live and let live, but this: I don't particularly find writing function(Arg) (as opposed to Arg.function) hard, at all - and I *certainly* don't want to pay *any* price in terms of added complexity or performance degradation for object oriented ish semantics or features. That's me though. I know a lot of people really like their language features and thank goodness we have options! _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED[mailto:erlang-questions@REDACTED] http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] From michal@REDACTED Sat Jan 14 20:56:00 2017 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Sat, 14 Jan 2017 20:56:00 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: On 14 Jan 2017, 17:00 +0100, Ilya Khaprov , wrote: > >> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? > > Why Elixir implements Unicode in Elixir? You have to rewrite it anyway. Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. BEAM is wonderful at optimising bitstring pattern matching, so the resulting code is quite efficient and does not need NIFs. The whole implementation is also about 500 LoC?https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex, so it's not a monstrosity and the code is quite readable. On 14 Jan 2017, 16:53 +0100, Oliver Korpilla , wrote: > Could the Unicode support in elixir serve as a starting point? > > https://hexdocs.pm/elixir/1.3.3/String.html#content > > String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: > > "For example, the codepoint ??? is two bytes: > > iex> byte_size("?") > 2" > > Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... All the functions in the Elixir's String module are unicode aware. There are functions for unicode normalisation (both nfd and nfc) and checking equivalency (e.g. "?" can be a single code point, or two code points - the letter "e" and the combining diacritic acute). String.length/1 works in terms of unicode graphemes (multiple codepoints can compose into a single grapheme): iex> String.length("?") 1 On 14 Jan 2017, 16:30 +0100, Benoit Chesneau , wrote: > someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS:?https://github.com/erlang-unicode/i18n icu is problematic in the fact that it works mostly with UTF-16. Micha? -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Sat Jan 14 21:12:41 2017 From: g@REDACTED (Garrett Smith) Date: Sat, 14 Jan 2017 14:12:41 -0600 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: On Sat, Jan 14, 2017 at 11:08 AM, John Doe wrote: >>string:to_upper("hello"). > > This function works for ASCII only. That's the problem. MICHA? = > string:to_upper("Micha?"). Neither does the string module work on binaries, which I've found inconvenient. Fortunately you can just use an alternative function/module. Sometimes I have to write my own function, which is also inconvenient. Modifying built-in type behavior (the vaunted Ruby string, but also Python, etc.) is super extremely inconvenient. From ilya.khaprov@REDACTED Sat Jan 14 21:19:25 2017 From: ilya.khaprov@REDACTED (Ilya Khaprov) Date: Sat, 14 Jan 2017 20:19:25 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> , Message-ID: >> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. Code generation. Can be done without macros, just generating Erlang files. So everyone in BEAM community can use. That is my point. ________________________________ From: Micha? Muska?a Sent: Saturday, January 14, 2017 22:56 To: Oliver Korpilla; Ilya Khaprov Cc: Erlang Questions Subject: RE: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead On 14 Jan 2017, 17:00 +0100, Ilya Khaprov , wrote: >> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? Why Elixir implements Unicode in Elixir? You have to rewrite it anyway. Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. BEAM is wonderful at optimising bitstring pattern matching, so the resulting code is quite efficient and does not need NIFs. The whole implementation is also about 500 LoC https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex, so it's not a monstrosity and the code is quite readable. [https://avatars2.githubusercontent.com/u/1481354?v=3&s=400] elixir/unicode.ex at master * elixir-lang/elixir * GitHub github.com elixir - Elixir is a dynamic, functional language designed for building scalable and maintainable applications On 14 Jan 2017, 16:53 +0100, Oliver Korpilla , wrote: Could the Unicode support in elixir serve as a starting point? https://hexdocs.pm/elixir/1.3.3/String.html#content String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: "For example, the codepoint "?" is two bytes: iex> byte_size("?") 2" Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... All the functions in the Elixir's String module are unicode aware. There are functions for unicode normalisation (both nfd and nfc) and checking equivalency (e.g. "?" can be a single code point, or two code points - the letter "e" and the combining diacritic acute). String.length/1 works in terms of unicode graphemes (multiple codepoints can compose into a single grapheme): iex> String.length("?") 1 On 14 Jan 2017, 16:30 +0100, Benoit Chesneau , wrote: someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS: https://github.com/erlang-unicode/i18n [https://avatars0.githubusercontent.com/u/2264078?v=3&s=400] GitHub - erlang-unicode/i18n: icu nif: international ... github.com i18n - icu nif: international components for unicode from Erlang (unicode, date, string, number, format, locale, localization, transliteration, icu4e) icu is problematic in the fact that it works mostly with UTF-16. Micha? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Sat Jan 14 21:25:09 2017 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Sat, 14 Jan 2017 21:25:09 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> , , Message-ID: Hello, Ilya. ? Erlang community can address all its own problems if it wants. You can do so if you have the means and if you want.? ? elixir developers made their own choices and I use elixir every day as my preferred BEAM language, not Erlang. I come here for BEAM and OTP knowledge. ? Besides, if you really want, you can write part of your application in elixir, benefit from its feature set, and code the rest in Erlang. BEAM and rebar support that. ? Which means: Everybody in BEAM community can _already_ use.? ? Oliver ? Gesendet:?Samstag, 14. Januar 2017 um 21:19 Uhr Von:?"Ilya Khaprov" An:?"Micha? Muska?a" , "Oliver Korpilla" Cc:?"Erlang Questions" Betreff:?Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead ? >> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. ? Code generation. Can be done without macros, just generating Erlang files. So everyone in BEAM community can use. That is my point. ? ------------------------------------------------------------ From: Micha? Muska?a Sent: Saturday, January 14, 2017 22:56 To: Oliver Korpilla; Ilya Khaprov Cc: Erlang Questions Subject: RE: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead ? On 14 Jan 2017, 17:00 +0100, Ilya Khaprov , wrote: ? >> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? ? Why Elixir implements Unicode in Elixir? You have to rewrite it anyway.? ? Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. BEAM is wonderful at optimising bitstring pattern matching, so the resulting code is quite efficient and does not need NIFs. The whole implementation is also about 500 LoC?https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex, so it's not a monstrosity and the code is quite readable. [https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex] ? elixir/unicode.ex at master ? elixir-lang/elixir ? GitHub[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex] github.com elixir - Elixir is a dynamic, functional language designed for building scalable and maintainable applications? ? On 14 Jan 2017, 16:53 +0100, Oliver Korpilla , wrote:Could the Unicode support in elixir serve as a starting point? https://hexdocs.pm/elixir/1.3.3/String.html#content String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: "For example, the codepoint ??? is two bytes: iex> byte_size("?") 2" Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... ? All the functions in the Elixir's String module are unicode aware. There are functions for unicode normalisation (both nfd and nfc) and checking equivalency (e.g. "?" can be a single code point, or two code points - the letter "e" and the combining diacritic acute). String.length/1 works in terms of unicode graphemes (multiple codepoints can compose into a single grapheme): ? iex> String.length("?") 1 ? On 14 Jan 2017, 16:30 +0100, Benoit Chesneau , wrote: someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS:?https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n] [https://github.com/erlang-unicode/i18n] ? GitHub - erlang-unicode/i18n: icu nif: international ...[https://github.com/erlang-unicode/i18n] github.com i18n - icu nif: international components for unicode from Erlang (unicode, date, string, number, format, locale, localization, transliteration, icu4e) ? icu is problematic in the fact that it works mostly with UTF-16. ? Micha? From t@REDACTED Sat Jan 14 21:33:46 2017 From: t@REDACTED (Tristan Sloughter) Date: Sat, 14 Jan 2017 12:33:46 -0800 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: <1484426026.4079769.847834608.18B5C3C9@webmail.messagingengine.com> Or better yet, a bif :) -- Tristan Sloughter "I am not a crackpot" - Abe Simpson t@REDACTED On Sat, Jan 14, 2017, at 08:06 AM, Benoit Chesneau wrote: > > > On Sat, Jan 14, 2017 at 4:53 PM Oliver Korpilla > wrote: >> Could the Unicode support in elixir serve as a starting point? >> >> https://hexdocs.pm/elixir/1.3.3/String.html#content >> >> String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And >> a lot of effort seems have gone in scenarios like this: >> >> "For example, the codepoint ??? is two bytes: >> >> iex> byte_size("?") >> 2" >> >> Given that both Erlang and elixir are implemented on top of BEAM, >> the wheel might not need reinventing? I know engineers and >> programmers love inventing stuff, and this discussion seems to point >> in that direction, but... >> >> Cheers, >> Oliver >> > > If I remember correctly the unicode support of Elixir is written in > elixir and data come from the unicode/icu projects. data resources > (codepoints and so on ) are compiled as beam. (I do the dame in my > idna lib). > The work may be simpler in using/wrting a nif over the well supported > ICU lib thoug. I'm curious about the reasonning that conducted to the > current implementation in elixir. > - benoit > > >> >> >> Gesendet: Freitag, 13. Januar 2017 um 23:34 Uhr >> Von: "Micha? Muska?a" >> An: "Richard A. O'Keefe" , "Steve Davis" >> , g@REDACTED, "Jesper Louis Andersen" >> >> Cc: "Erlang Questions" >> Betreff: Re: [erlang-questions] Erlang basic doubts about String, >> message passing and context switching overhead >> >> I fully agree there are no languages that deal with strings >> perfectly. That said there are those that are better at it and >> those that aren't so good. A language, where I need to look for a >> library to upcase or downcase my own name, fits into the second >> group in my book. >> >> Micha?. >> On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen >> , wrote: >> >> Richard is indeed right, depending on what your definition of >> "String" is. >> If a "String" is "An array of characters from some alphabet", then >> you need to take into account Strings are Unicode codepoints in >> practice. This is also the most precise definition from a technical >> point of view. >> When I wrote my post, I was--probably incorrectly--assuming the >> older notion of a "String" where the representation is either ASCII >> or something like ISO-8859-15. In this case, a string coincides >> with a stream of bytes. >> Data needs parsing. A lot of data comes in as some kind of stringy >> representation: UTF-8, byte array (binary), and so on. >> And of course, that isn't the whole story, since there are examples >> of input which are not string-like in their forms. >> >> >> On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe >> wrote: >> >> On 13/01/17 8:56 AM, Jesper Louis Andersen wrote: >> > Strings are really just streams of bytes. >> >> That was true a long time ago. Maybe. >> But it isn't anywhere near accurate as a description >> of Unicode: >> - Unicode is made of 21-bit code points, not bytes. >> - Most possible code points are not defined. >> - Some of those that are defined are defined as >> "it is illegal to use this". >> - Unicode sequences have *structure*; it is simply >> not the case that every sequence of allowable >> Unicode code points is a legal Unicode string. >> - As a special case of that, if s is a non-empty >> valid Unicode string, it is not true that every >> substring of s is a valid Unicode string. >> >> In case you were thinking of UTF-8, not all byte >> sequences are valid UTF-8. >> >> Byte streams are as important as you say, but it's >> really hard to see the software for a radar or a >> radio telescope as processing strings... >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________ >> erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions[ >> http://erlang.org/mailman/listinfo/erlang-questions][1] >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Links: 1. http://erlang.org/mailman/listinfo/erlang-questions%5B http://erlang.org/mailman/listinfo/erlang-questions%5D -------------- next part -------------- An HTML attachment was scrubbed... URL: From dszoboszlay@REDACTED Sat Jan 14 21:47:07 2017 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Sat, 14 Jan 2017 20:47:07 +0000 Subject: [erlang-questions] How can I release beam process memory? In-Reply-To: References: Message-ID: Hi Jack, I guess the 9 GB is lost due to memory fragmentation. Erlang allocates memory in large chunks called carriers from the OS, then places the blocks your program actually needs on these carriers. A carrier can only be returned to the OS once all the blocks on it have been freed (and even then, the memory allocator may decide to keep it around for a while in case more memory is needed). You can check with recon_alloc how much unused memory is lost due to fragmentation in the various allocators. The bad news is that you cannot defragment the carriers, and if the selected memory allocator strategy doesn't work well for your application, you cannot change it either without restarting the emulator. However, if the memory is wasted in the eheap_alloc, you may try to force a GC on all processes a couple of times. As the GC copies the memory, it will allocate new blocks and free up the old heap blocks. So there's a chance the allocators can compact the blocks together on fewer segments. But that's just a guess, it may or may not work at all. Cheers, Daniel On Sat, 14 Jan 2017 at 08:04 Jack Tang wrote: > Hello list, > > I run one Erlang application on Debian server and today I find the beam > process consumes around 35G memory by `top` command. > > ``` > KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers > KiB Swap: 0 total, 0 used, 0 free. 18684864 cached Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp > ``` > > I connect to the Erlang application using remote shell and find the > mem-leaked supervisor tree and run gc on the whole tree. Code looks like > blow: > > ``` > lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) > end, supervisor:which_children(some_thing_sup)). > ``` > > and erlang:memory() decreases from 32G to 23G. > ``` > [{total,22982011544}, > {processes,12884182336}, > {processes_used,12884170336}, > {system,10097829208}, > {atom,13828705}, > {atom_used,13796692}, > {binary,170530288}, > {code,16450626}, > {ets,9637717576}] > ``` > > However, when I input `top` command, the beam process still takes 35G > memory. What can I do to release the 9G memory? Thanks > > BR > -Jack > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sat Jan 14 22:32:32 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Sat, 14 Jan 2017 22:32:32 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: On 01/14/2017 09:25 PM, Oliver Korpilla wrote: > Erlang community can address all its own problems if it wants. You can do so if you have the means and if you want. Sounds like great cooperation. > elixir developers made their own choices and I use elixir every day as my preferred BEAM language, not Erlang. I come here for BEAM and OTP knowledge. Completely unrelated to making the Unicode code available for all BEAM languages without the obnoxious dependency. > Besides, if you really want, you can write part of your application in elixir, benefit from its feature set, and code the rest in Erlang. BEAM and rebar support that. > > Which means: Everybody in BEAM community can _already_ use. No, everybody can't use it. For example, if the httpbis working group were to add Unicode HTTP headers[1], I would need to find a way to make it work that does not involve Elixir, because I can't force that big a dependency on everyone. Saying "it's there therefore you can use it" is incredibly naive. [1] It's being discussed. > Oliver > > > Gesendet: Samstag, 14. Januar 2017 um 21:19 Uhr > Von: "Ilya Khaprov" > An: "Micha? Muska?a" , "Oliver Korpilla" > Cc: "Erlang Questions" > Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead > > > >>> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. > > Code generation. Can be done without macros, just generating Erlang files. So everyone in BEAM community can use. That is my point. > > ------------------------------------------------------------ > > From: Micha? Muska?a > Sent: Saturday, January 14, 2017 22:56 > To: Oliver Korpilla; Ilya Khaprov > Cc: Erlang Questions > Subject: RE: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead > > > On 14 Jan 2017, 17:00 +0100, Ilya Khaprov , wrote: > > >>> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? > > Why Elixir implements Unicode in Elixir? You have to rewrite it anyway. > > > Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. BEAM is wonderful at optimising bitstring pattern matching, so the resulting code is quite efficient and does not need NIFs. The whole implementation is also about 500 LoC https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex, so it's not a monstrosity and the code is quite readable. > > [https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex] > > elixir/unicode.ex at master ? elixir-lang/elixir ? GitHub[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex] > github.com > elixir - Elixir is a dynamic, functional language designed for building scalable and maintainable applications > > On 14 Jan 2017, 16:53 +0100, Oliver Korpilla , wrote:Could the Unicode support in elixir serve as a starting point? > > https://hexdocs.pm/elixir/1.3.3/String.html#content > > String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: > > "For example, the codepoint ??? is two bytes: > > iex> byte_size("?") > 2" > > Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... > > All the functions in the Elixir's String module are unicode aware. There are functions for unicode normalisation (both nfd and nfc) and checking equivalency (e.g. "?" can be a single code point, or two code points - the letter "e" and the combining diacritic acute). String.length/1 works in terms of unicode graphemes (multiple codepoints can compose into a single grapheme): > > iex> String.length("?") > 1 > > On 14 Jan 2017, 16:30 +0100, Benoit Chesneau , wrote: > someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS: https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n] > > [https://github.com/erlang-unicode/i18n] > > GitHub - erlang-unicode/i18n: icu nif: international ...[https://github.com/erlang-unicode/i18n] > github.com > i18n - icu nif: international components for unicode from Erlang (unicode, date, string, number, format, locale, localization, transliteration, icu4e) > > icu is problematic in the fact that it works mostly with UTF-16. > > Micha? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From Oliver.Korpilla@REDACTED Sat Jan 14 22:46:17 2017 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Sat, 14 Jan 2017 22:46:17 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> , Message-ID: Hello, Loic. Great cooperation? More like a reaction to the attitude I get from you and Ilya. I actually do believe the Erlang can and has to solve its own problems. As you see, behind Erlang there is a certain set of ideas, principles, and design decisions, and behind elixir is another. To demand of somebody else to solve your problems in your preferred language... is offensive to me. So I reacted. Some people prefer Erlang, some prefer elixir. Telling people that they should have implemented a solution in Erlang to make it available to "the community" describes precisely a mindset where you say "my community is fine, what silly thing are you doing over there?" And that is exactly how cooperation breaks down. Calling me naive is similarly offensive. Just because you do not want to do something, don't resort to name-calling. Nobody said that some group _should_ use elixir, but the possibility exists. Nothing more was ever said. To infer from your own projections I am naive is rich. To make it clear, I do not speak in any capacity for the elixir community. I am just a user trying to understand the work of others, which I admire, in BOTH communities. I'm pretty sure you don't speak for Erlang community either. Oliver ? ? Gesendet:?Samstag, 14. Januar 2017 um 22:32 Uhr Von:?"Lo?c Hoguin" An:?"Oliver Korpilla" , "Ilya Khaprov" Cc:?"Erlang Questions" Betreff:?Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead On 01/14/2017 09:25 PM, Oliver Korpilla wrote: > Erlang community can address all its own problems if it wants. You can do so if you have the means and if you want. Sounds like great cooperation. > elixir developers made their own choices and I use elixir every day as my preferred BEAM language, not Erlang. I come here for BEAM and OTP knowledge. Completely unrelated to making the Unicode code available for all BEAM languages without the obnoxious dependency. > Besides, if you really want, you can write part of your application in elixir, benefit from its feature set, and code the rest in Erlang. BEAM and rebar support that. > > Which means: Everybody in BEAM community can _already_ use. No, everybody can't use it. For example, if the httpbis working group were to add Unicode HTTP headers[1], I would need to find a way to make it work that does not involve Elixir, because I can't force that big a dependency on everyone. Saying "it's there therefore you can use it" is incredibly naive. [1] It's being discussed. > Oliver > > > Gesendet: Samstag, 14. Januar 2017 um 21:19 Uhr > Von: "Ilya Khaprov" > An: "Micha? Muska?a" , "Oliver Korpilla" > Cc: "Erlang Questions" > Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead > > > >>> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. > > Code generation. Can be done without macros, just generating Erlang files. So everyone in BEAM community can use. That is my point. > > ------------------------------------------------------------ > > From: Micha? Muska?a > Sent: Saturday, January 14, 2017 22:56 > To: Oliver Korpilla; Ilya Khaprov > Cc: Erlang Questions > Subject: RE: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead > > > On 14 Jan 2017, 17:00 +0100, Ilya Khaprov , wrote: > > >>> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? > > Why Elixir implements Unicode in Elixir? You have to rewrite it anyway. > > > Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. BEAM is wonderful at optimising bitstring pattern matching, so the resulting code is quite efficient and does not need NIFs. The whole implementation is also about 500 LoC https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex, so it's not a monstrosity and the code is quite readable. > > [https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex]] > > elixir/unicode.ex at master ? elixir-lang/elixir ? GitHub[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex]] > github.com > elixir - Elixir is a dynamic, functional language designed for building scalable and maintainable applications > > On 14 Jan 2017, 16:53 +0100, Oliver Korpilla , wrote:Could the Unicode support in elixir serve as a starting point? > > https://hexdocs.pm/elixir/1.3.3/String.html#content[https://hexdocs.pm/elixir/1.3.3/String.html#content] > > String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: > > "For example, the codepoint ??? is two bytes: > > iex> byte_size("?") > 2" > > Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... > > All the functions in the Elixir's String module are unicode aware. There are functions for unicode normalisation (both nfd and nfc) and checking equivalency (e.g. "?" can be a single code point, or two code points - the letter "e" and the combining diacritic acute). String.length/1 works in terms of unicode graphemes (multiple codepoints can compose into a single grapheme): > > iex> String.length("?") > 1 > > On 14 Jan 2017, 16:30 +0100, Benoit Chesneau , wrote: > someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS: https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n][https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] > > [https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] > > GitHub - erlang-unicode/i18n: icu nif: international ...[https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] > github.com > i18n - icu nif: international components for unicode from Erlang (unicode, date, string, number, format, locale, localization, transliteration, icu4e) > > icu is problematic in the fact that it works mostly with UTF-16. > > Micha? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] > -- Lo?c Hoguin https://ninenines.eu[https://ninenines.eu] From essen@REDACTED Sat Jan 14 23:21:19 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Sat, 14 Jan 2017 23:21:19 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: <43612f48-6f93-c6c0-d794-931f29db6a7f@ninenines.eu> What attitude? He's not wrong. It will need to be implemented in Erlang eventually. I'm not saying there won't be any cooperation when that happens, I'm sure there will be, Jos? has been doing great work contributing. My problem is with how you put it. You make it sound like this is a problem only for Erlang people only. You realize Elixir is implemented on top of Erlang, right? And that it'd most likely end up using the Erlang implementation if it makes it inside OTP, instead of duplicating the work? I'm also not saying that it was the wrong call to implement it *for* Elixir first. It clearly was the right call. But it probably could have benefited a lot more people with minimum extra efforts. And just so we're clear, you should feel offended that I called your comment naive. Because it really was. There's plenty of reasons why one can't use Elixir code. I gave you one. There's more. It's okay though, we're all naive about many, many things. Anyway. Elixir has some Unicode support. Great for the people that can use it! Everyone else still has a problem. Let's put down the pitchforks and torches and discuss that. Cheers, On 01/14/2017 10:46 PM, Oliver Korpilla wrote: > Hello, Loic. > > Great cooperation? More like a reaction to the attitude I get from you and Ilya. > > I actually do believe the Erlang can and has to solve its own problems. As you see, behind Erlang there is a certain set of ideas, principles, and design decisions, and behind elixir is another. To demand of somebody else to solve your problems in your preferred language... is offensive to me. So I reacted. > > Some people prefer Erlang, some prefer elixir. Telling people that they should have implemented a solution in Erlang to make it available to "the community" describes precisely a mindset where you say "my community is fine, what silly thing are you doing over there?" And that is exactly how cooperation breaks down. > > Calling me naive is similarly offensive. Just because you do not want to do something, don't resort to name-calling. Nobody said that some group _should_ use elixir, but the possibility exists. Nothing more was ever said. To infer from your own projections I am naive is rich. > > To make it clear, I do not speak in any capacity for the elixir community. I am just a user trying to understand the work of others, which I admire, in BOTH communities. > > I'm pretty sure you don't speak for Erlang community either. > > Oliver > > > > Gesendet: Samstag, 14. Januar 2017 um 22:32 Uhr > Von: "Lo?c Hoguin" > An: "Oliver Korpilla" , "Ilya Khaprov" > Cc: "Erlang Questions" > Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead > On 01/14/2017 09:25 PM, Oliver Korpilla wrote: >> Erlang community can address all its own problems if it wants. You can do so if you have the means and if you want. > > Sounds like great cooperation. > >> elixir developers made their own choices and I use elixir every day as my preferred BEAM language, not Erlang. I come here for BEAM and OTP knowledge. > > Completely unrelated to making the Unicode code available for all BEAM > languages without the obnoxious dependency. > >> Besides, if you really want, you can write part of your application in elixir, benefit from its feature set, and code the rest in Erlang. BEAM and rebar support that. >> >> Which means: Everybody in BEAM community can _already_ use. > > No, everybody can't use it. > > For example, if the httpbis working group were to add Unicode HTTP > headers[1], I would need to find a way to make it work that does not > involve Elixir, because I can't force that big a dependency on everyone. > > Saying "it's there therefore you can use it" is incredibly naive. > > [1] It's being discussed. > >> Oliver >> >> >> Gesendet: Samstag, 14. Januar 2017 um 21:19 Uhr >> Von: "Ilya Khaprov" >> An: "Micha? Muska?a" , "Oliver Korpilla" >> Cc: "Erlang Questions" >> Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead >> >> >> >>>> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. >> >> Code generation. Can be done without macros, just generating Erlang files. So everyone in BEAM community can use. That is my point. >> >> ------------------------------------------------------------ >> >> From: Micha? Muska?a >> Sent: Saturday, January 14, 2017 22:56 >> To: Oliver Korpilla; Ilya Khaprov >> Cc: Erlang Questions >> Subject: RE: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead >> >> >> On 14 Jan 2017, 17:00 +0100, Ilya Khaprov , wrote: >> >> >>>> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? >> >> Why Elixir implements Unicode in Elixir? You have to rewrite it anyway. >> >> >> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. BEAM is wonderful at optimising bitstring pattern matching, so the resulting code is quite efficient and does not need NIFs. The whole implementation is also about 500 LoC https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex, so it's not a monstrosity and the code is quite readable. >> >> [https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex]] >> >> elixir/unicode.ex at master ? elixir-lang/elixir ? GitHub[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex]] >> github.com >> elixir - Elixir is a dynamic, functional language designed for building scalable and maintainable applications >> >> On 14 Jan 2017, 16:53 +0100, Oliver Korpilla , wrote:Could the Unicode support in elixir serve as a starting point? >> >> https://hexdocs.pm/elixir/1.3.3/String.html#content[https://hexdocs.pm/elixir/1.3.3/String.html#content] >> >> String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: >> >> "For example, the codepoint ??? is two bytes: >> >> iex> byte_size("?") >> 2" >> >> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... >> >> All the functions in the Elixir's String module are unicode aware. There are functions for unicode normalisation (both nfd and nfc) and checking equivalency (e.g. "?" can be a single code point, or two code points - the letter "e" and the combining diacritic acute). String.length/1 works in terms of unicode graphemes (multiple codepoints can compose into a single grapheme): >> >> iex> String.length("?") >> 1 >> >> On 14 Jan 2017, 16:30 +0100, Benoit Chesneau , wrote: >> someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS: https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n][https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> >> [https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> >> GitHub - erlang-unicode/i18n: icu nif: international ...[https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> github.com >> i18n - icu nif: international components for unicode from Erlang (unicode, date, string, number, format, locale, localization, transliteration, icu4e) >> >> icu is problematic in the fact that it works mostly with UTF-16. >> >> Micha? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] >> > > -- > Lo?c Hoguin > https://ninenines.eu[https://ninenines.eu] > -- Lo?c Hoguin https://ninenines.eu From list1@REDACTED Sat Jan 14 23:54:59 2017 From: list1@REDACTED (Grzegorz Junka) Date: Sat, 14 Jan 2017 22:54:59 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: <7cbc41cf-7d19-ad8e-c6f1-59e9fa32f77b@gjunka.com> Hello Olivier, Initially I didn't get your email, but then I thought that you are responding like if Loic was complaining about Elixir not implementing something in Erlang. I understood his email completely differently. To me all that Loic was saying is that Erlang shouldn't need to rely on Elixir because it's not always possible to depend an Erlang application on Elixir. Thinking that we can simply reuse C or Elixir libraries is naive, which is a general opinion, not necessarily related to your post (since you weren't the first one to propose to reuse libraries from other languages). The sentence excerpt is just a context for his opinion. Erlang wasn't designed with text processing in mind and still isn't that its primary objective. Only in recent years Erlang started to be used to implement non-telephony related applications and there are some areas which need to catch up, unicode including. Besides, porting 500 lines of code from Elixir to Erlang shouldn't be that difficult? Grzegorz On 14/01/2017 21:46, Oliver Korpilla wrote: > Hello, Loic. > > Great cooperation? More like a reaction to the attitude I get from you and Ilya. > > I actually do believe the Erlang can and has to solve its own problems. As you see, behind Erlang there is a certain set of ideas, principles, and design decisions, and behind elixir is another. To demand of somebody else to solve your problems in your preferred language... is offensive to me. So I reacted. > > Some people prefer Erlang, some prefer elixir. Telling people that they should have implemented a solution in Erlang to make it available to "the community" describes precisely a mindset where you say "my community is fine, what silly thing are you doing over there?" And that is exactly how cooperation breaks down. > > Calling me naive is similarly offensive. Just because you do not want to do something, don't resort to name-calling. Nobody said that some group _should_ use elixir, but the possibility exists. Nothing more was ever said. To infer from your own projections I am naive is rich. > > To make it clear, I do not speak in any capacity for the elixir community. I am just a user trying to understand the work of others, which I admire, in BOTH communities. > > I'm pretty sure you don't speak for Erlang community either. > > Oliver > > > > Gesendet: Samstag, 14. Januar 2017 um 22:32 Uhr > Von: "Lo?c Hoguin" > An: "Oliver Korpilla" , "Ilya Khaprov" > Cc: "Erlang Questions" > Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead > On 01/14/2017 09:25 PM, Oliver Korpilla wrote: >> Erlang community can address all its own problems if it wants. You can do so if you have the means and if you want. > Sounds like great cooperation. > >> elixir developers made their own choices and I use elixir every day as my preferred BEAM language, not Erlang. I come here for BEAM and OTP knowledge. > Completely unrelated to making the Unicode code available for all BEAM > languages without the obnoxious dependency. > >> Besides, if you really want, you can write part of your application in elixir, benefit from its feature set, and code the rest in Erlang. BEAM and rebar support that. >> >> Which means: Everybody in BEAM community can _already_ use. > No, everybody can't use it. > > For example, if the httpbis working group were to add Unicode HTTP > headers[1], I would need to find a way to make it work that does not > involve Elixir, because I can't force that big a dependency on everyone. > > Saying "it's there therefore you can use it" is incredibly naive. > > [1] It's being discussed. > >> Oliver >> >> >> Gesendet: Samstag, 14. Januar 2017 um 21:19 Uhr >> Von: "Ilya Khaprov" >> An: "Micha? Muska?a" , "Oliver Korpilla" >> Cc: "Erlang Questions" >> Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead >> >> >> >>>> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. >> Code generation. Can be done without macros, just generating Erlang files. So everyone in BEAM community can use. That is my point. >> >> ------------------------------------------------------------ >> >> From: Micha? Muska?a >> Sent: Saturday, January 14, 2017 22:56 >> To: Oliver Korpilla; Ilya Khaprov >> Cc: Erlang Questions >> Subject: RE: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead >> >> >> On 14 Jan 2017, 17:00 +0100, Ilya Khaprov , wrote: >> >> >>>> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? >> Why Elixir implements Unicode in Elixir? You have to rewrite it anyway. >> >> >> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. BEAM is wonderful at optimising bitstring pattern matching, so the resulting code is quite efficient and does not need NIFs. The whole implementation is also about 500 LoC https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex, so it's not a monstrosity and the code is quite readable. >> >> [https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex]] >> >> elixir/unicode.ex at master ? elixir-lang/elixir ? GitHub[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex]] >> github.com >> elixir - Elixir is a dynamic, functional language designed for building scalable and maintainable applications >> >> On 14 Jan 2017, 16:53 +0100, Oliver Korpilla , wrote:Could the Unicode support in elixir serve as a starting point? >> >> https://hexdocs.pm/elixir/1.3.3/String.html#content[https://hexdocs.pm/elixir/1.3.3/String.html#content] >> >> String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: >> >> "For example, the codepoint ??? is two bytes: >> >> iex> byte_size("?") >> 2" >> >> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... >> >> All the functions in the Elixir's String module are unicode aware. There are functions for unicode normalisation (both nfd and nfc) and checking equivalency (e.g. "?" can be a single code point, or two code points - the letter "e" and the combining diacritic acute). String.length/1 works in terms of unicode graphemes (multiple codepoints can compose into a single grapheme): >> >> iex> String.length("?") >> 1 >> >> On 14 Jan 2017, 16:30 +0100, Benoit Chesneau , wrote: >> someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS: https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n][https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> >> [https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> >> GitHub - erlang-unicode/i18n: icu nif: international ...[https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> github.com >> i18n - icu nif: international components for unicode from Erlang (unicode, date, string, number, format, locale, localization, transliteration, icu4e) >> >> icu is problematic in the fact that it works mostly with UTF-16. >> >> Micha? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] >> > -- > Lo?c Hoguin > https://ninenines.eu[https://ninenines.eu] > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From frank.muller.erl@REDACTED Sun Jan 15 00:03:52 2017 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 14 Jan 2017 23:03:52 +0000 Subject: [erlang-questions] How can I release beam process memory? In-Reply-To: References: Message-ID: I'm very interested in that too. Last time I asked for help only @Max Lapshin was kind enough to share some feedbacks (today's Daniel). I would appreciate if the OTP team can teach us how to debug VM's mem issues, and how pick the right allocator. I'm aware of the excellent recon module. But to be honest, I was never be able to successfully use it to track my mem issue. Maybe not using it correctly. Thanks in advance. Best, /Frank Le sam. 14 janv. 2017 ? 21:47, D?niel Szoboszlay a ?crit : > Hi Jack, > > I guess the 9 GB is lost due to memory fragmentation. Erlang allocates > memory in large chunks called carriers from the OS, then places the blocks > your program actually needs on these carriers. A carrier can only be > returned to the OS once all the blocks on it have been freed (and even > then, the memory allocator may decide to keep it around for a while in case > more memory is needed). > > You can check with recon_alloc > how much unused memory is > lost due to fragmentation in the various allocators. > > The bad news is that you cannot defragment the carriers, and if the > selected memory allocator strategy doesn't work well for your application, > you cannot change it either without restarting the emulator. > > However, if the memory is wasted in the eheap_alloc, you may try to force > a GC on all processes a couple of times. As the GC copies the memory, it > will allocate new blocks and free up the old heap blocks. So there's a > chance the allocators can compact the blocks together on fewer segments. > But that's just a guess, it may or may not work at all. > > Cheers, > Daniel > > On Sat, 14 Jan 2017 at 08:04 Jack Tang wrote: > > Hello list, > > I run one Erlang application on Debian server and today I find the beam > process consumes around 35G memory by `top` command. > > ``` > KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers > KiB Swap: 0 total, 0 used, 0 free. 18684864 cached Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp > ``` > > I connect to the Erlang application using remote shell and find the > mem-leaked supervisor tree and run gc on the whole tree. Code looks like > blow: > > ``` > lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) > end, supervisor:which_children(some_thing_sup)). > ``` > > and erlang:memory() decreases from 32G to 23G. > ``` > [{total,22982011544}, > {processes,12884182336}, > {processes_used,12884170336}, > {system,10097829208}, > {atom,13828705}, > {atom_used,13796692}, > {binary,170530288}, > {code,16450626}, > {ets,9637717576}] > ``` > > However, when I input `top` command, the beam process still takes 35G > memory. What can I do to release the 9G memory? Thanks > > BR > -Jack > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Sun Jan 15 00:07:15 2017 From: t@REDACTED (Tristan Sloughter) Date: Sat, 14 Jan 2017 15:07:15 -0800 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <7cbc41cf-7d19-ad8e-c6f1-59e9fa32f77b@gjunka.com> References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> <7cbc41cf-7d19-ad8e-c6f1-59e9fa32f77b@gjunka.com> Message-ID: <1484435235.4109050.847908288.26B4F9FF@webmail.messagingengine.com> > Besides, porting 500 lines of code from Elixir to Erlang shouldn't be that difficult? It isn't really 500 lines of code. It uses macros. So it is 500 lines of code that writes lines of code. From ilya.khaprov@REDACTED Sun Jan 15 00:26:05 2017 From: ilya.khaprov@REDACTED (Ilya Khaprov) Date: Sat, 14 Jan 2017 23:26:05 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <7cbc41cf-7d19-ad8e-c6f1-59e9fa32f77b@gjunka.com> References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> , <7cbc41cf-7d19-ad8e-c6f1-59e9fa32f77b@gjunka.com> Message-ID: It was likely me who triggered him. But I only noted that while it?s not reinventing the wheel it?s still a rewrite. And yes, I?m sure most of the Elixir?s ?stdlib? could be implemented in Erlang for everyone?s benefit (BEAM communality is too small to afford fragmenting). Compare using Erlang from Elixir and Elixir from Erlang. But I fully respect people?s choice of the language. No need to be offended. >> Only in recent years Erlang started to be used to implement non-telephony related applications For example, Ejabberd started in 2002 and released in 2005. >> porting 500 lines of code from Elixir to Erlang shouldn't be that difficult As Tristan noted, it?s a macro-heavy code. And someone has to make sure it?s really correct. Regards. From: Grzegorz Junka Sent: Sunday, January 15, 2017 01:55 AM To: erlang-questions@REDACTED Subject: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead Hello Olivier, Initially I didn't get your email, but then I thought that you are responding like if Loic was complaining about Elixir not implementing something in Erlang. I understood his email completely differently. To me all that Loic was saying is that Erlang shouldn't need to rely on Elixir because it's not always possible to depend an Erlang application on Elixir. Thinking that we can simply reuse C or Elixir libraries is naive, which is a general opinion, not necessarily related to your post (since you weren't the first one to propose to reuse libraries from other languages). The sentence excerpt is just a context for his opinion. Erlang wasn't designed with text processing in mind and still isn't that its primary objective. Only in recent years Erlang started to be used to implement non-telephony related applications and there are some areas which need to catch up, unicode including. Besides, porting 500 lines of code from Elixir to Erlang shouldn't be that difficult? Grzegorz On 14/01/2017 21:46, Oliver Korpilla wrote: > Hello, Loic. > > Great cooperation? More like a reaction to the attitude I get from you and Ilya. > > I actually do believe the Erlang can and has to solve its own problems. As you see, behind Erlang there is a certain set of ideas, principles, and design decisions, and behind elixir is another. To demand of somebody else to solve your problems in your preferred language... is offensive to me. So I reacted. > > Some people prefer Erlang, some prefer elixir. Telling people that they should have implemented a solution in Erlang to make it available to "the community" describes precisely a mindset where you say "my community is fine, what silly thing are you doing over there?" And that is exactly how cooperation breaks down. > > Calling me naive is similarly offensive. Just because you do not want to do something, don't resort to name-calling. Nobody said that some group _should_ use elixir, but the possibility exists. Nothing more was ever said. To infer from your own projections I am naive is rich. > > To make it clear, I do not speak in any capacity for the elixir community. I am just a user trying to understand the work of others, which I admire, in BOTH communities. > > I'm pretty sure you don't speak for Erlang community either. > > Oliver > > > > Gesendet: Samstag, 14. Januar 2017 um 22:32 Uhr > Von: "Lo?c Hoguin" > An: "Oliver Korpilla" , "Ilya Khaprov" > Cc: "Erlang Questions" > Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead > On 01/14/2017 09:25 PM, Oliver Korpilla wrote: >> Erlang community can address all its own problems if it wants. You can do so if you have the means and if you want. > Sounds like great cooperation. > >> elixir developers made their own choices and I use elixir every day as my preferred BEAM language, not Erlang. I come here for BEAM and OTP knowledge. > Completely unrelated to making the Unicode code available for all BEAM > languages without the obnoxious dependency. > >> Besides, if you really want, you can write part of your application in elixir, benefit from its feature set, and code the rest in Erlang. BEAM and rebar support that. >> >> Which means: Everybody in BEAM community can _already_ use. > No, everybody can't use it. > > For example, if the httpbis working group were to add Unicode HTTP > headers[1], I would need to find a way to make it work that does not > involve Elixir, because I can't force that big a dependency on everyone. > > Saying "it's there therefore you can use it" is incredibly naive. > > [1] It's being discussed. > >> Oliver >> >> >> Gesendet: Samstag, 14. Januar 2017 um 21:19 Uhr >> Von: "Ilya Khaprov" >> An: "Micha? Muska?a" , "Oliver Korpilla" >> Cc: "Erlang Questions" >> Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead >> >> >> >>>> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. >> Code generation. Can be done without macros, just generating Erlang files. So everyone in BEAM community can use. That is my point. >> >> ------------------------------------------------------------ >> >> From: Micha? Muska?a >> Sent: Saturday, January 14, 2017 22:56 >> To: Oliver Korpilla; Ilya Khaprov >> Cc: Erlang Questions >> Subject: RE: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead >> >> >> On 14 Jan 2017, 17:00 +0100, Ilya Khaprov , wrote: >> >> >>>> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? >> Why Elixir implements Unicode in Elixir? You have to rewrite it anyway. >> >> >> Elixir takes the files directly from unicode distribution, and uses macros to compile them down into regular functions. BEAM is wonderful at optimising bitstring pattern matching, so the resulting code is quite efficient and does not need NIFs. The whole implementation is also about 500 LoC https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex, so it's not a monstrosity and the code is quite readable. >> >> [https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex]] >> >> elixir/unicode.ex at master ? elixir-lang/elixir ? GitHub[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex[https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex]] >> github.com >> elixir - Elixir is a dynamic, functional language designed for building scalable and maintainable applications >> >> On 14 Jan 2017, 16:53 +0100, Oliver Korpilla , wrote:Could the Unicode support in elixir serve as a starting point? >> >> https://hexdocs.pm/elixir/1.3.3/String.html#content[https://hexdocs.pm/elixir/1.3.3/String.html#content] >> >> String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this: >> >> "For example, the codepoint ??? is two bytes: >> >> iex> byte_size("?") >> 2" >> >> Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but... >> >> All the functions in the Elixir's String module are unicode aware. There are functions for unicode normalisation (both nfd and nfc) and checking equivalency (e.g. "?" can be a single code point, or two code points - the letter "e" and the combining diacritic acute). String.length/1 works in terms of unicode graphemes (multiple codepoints can compose into a single grapheme): >> >> iex> String.length("?") >> 1 >> >> On 14 Jan 2017, 16:30 +0100, Benoit Chesneau , wrote: >> someone has to write a good binding (non blocking) of icu :) I think it would be easier than reinventing the wheel. i18n [1] was a good start but looks abandoned these days. Also I disliked the load of resources at startup in ETS: https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n][https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> >> [https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> >> GitHub - erlang-unicode/i18n: icu nif: international ...[https://github.com/erlang-unicode/i18n[https://github.com/erlang-unicode/i18n]] >> github.com >> i18n - icu nif: international components for unicode from Erlang (unicode, date, string, number, format, locale, localization, transliteration, icu4e) >> >> icu is problematic in the fact that it works mostly with UTF-16. >> >> Micha? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] >> > -- > Lo?c Hoguin > https://ninenines.eu[https://ninenines.eu] > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sun Jan 15 03:25:25 2017 From: mjtruog@REDACTED (Michael Truog) Date: Sat, 14 Jan 2017 18:25:25 -0800 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: Message-ID: <587ADD95.6040109@gmail.com> ||This thread started to talk about the need for unicode functionality in Erlang and how it exists currently in Elixir but not in Erlang. I created a repository with the Elixir functions created as Erlang functions in an Erlang module as an example of what I want at https://github.com/okeuday/unicode_data/ The generated module (unicode_data) only includes functions from the first Erlang module contained in unicode.ex (Elixir.String.Unicode) though it does bring up some important topics: 1) Add the unicode version to the Erlang module version. The generated unicode_data module has a timestamp suffix, so we know both the unicode version and the timestamp when the unicode_data module was generated. 2) Use only lists, not binaries, to make sure all temporary data stays on the heap of the Erlang process. That should be best for performance, though I haven't proved that with any performance testing. I haven't added tests, though I have compared the unicode_data Erlang module, to the Elixir.String.Unicode module and the data looks correct. Mainly thought this would help the discussion. Best Regards, Michael On 01/10/2017 10:58 AM, Bhag Chandra wrote: > Hello, > > I have been coding in Erlang for 2 years. A wonderful language but not very big community, so I cant discuss my questions with programmers around me (Java, Python guys). I found out about this list today. > > I have some fundamental doubts about the Erlang. It would be great if someone can help me clarify them. > > > 1) "Strings in Erlang are internally treated as a list of integers of each character's ASCII values, this representation of string makes operations faster. For example, string concatenation is constant time operation in Erlang." Can someone explain why? > > 2) "It makes sense to use Erlang only where system's availability is very high". Is it not a very general requirement of most of the systems? Whatsapp to Google to FB to Amazon to Paypal to Barclays etc they all are high availability systems, so we can use Erlang in all of them? > > 3) "Every message which is sent to a process, goes to the mailbox of that process. When process is free, it consumes that message from mailbox". So how exactly does process ask from the mailbox for that message? Is there a mechanism in a process' memory which keeps polling its mailbox. I basically want to understand how message is sent from mailbox to my code in process. > > 4) We say that a message is passed from process A to process B by simply using a bang (!) character, but what happens behind the scenes to pass this message? Do both processes establish a tcp connection first and then pass message or what? > > 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) Mr. Armstrong is talking about the difference between the context switching overhead between OS threads and Erlang processes. He says, thread context switching is of order 700 words but Erlang process context switching is ... ? > I cant understand what he said, if someone could tell. > > > P.S. Please excuse for any grammatical errors, English is not my first language. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From alangingras@REDACTED Sat Jan 14 22:19:19 2017 From: alangingras@REDACTED (Alan Gingras) Date: Sat, 14 Jan 2017 16:19:19 -0500 Subject: [erlang-questions] Erlang ODBC Message-ID: <001e01d26eab$df454ed0$9dcfec70$@comcast.net> Fairly new to Erlang and trying to use ODBC to connect to an excel spreadsheet. Connect returns { ok, Ref } but any query seems to fail. I.e.: 12> odbc:sql_query( Ref, "SELECT * FROM Sheet1;" ). {error,{"42S02",-1305, "[Microsoft][ODBC Excel Driver] The Microsoft Access database engine could not find the object 'MyTable1'. Make sure the object exists and that you spell its name and the path name correctly. If 'MyTable1' is not a local object, check your network connection or contact the server administrator."}} Spreadsheet is a single sheet (Sheet1). Have also tried "SELECT * FROM [Sheet1$];" with same result. Have tried both without the ';' as well. Any suggestions, ideas or examples would be welcome. Using excel because I can download the data as a spreadsheet from the source website. Thanks for any help given. -------------- next part -------------- An HTML attachment was scrubbed... URL: From himars@REDACTED Sun Jan 15 07:09:41 2017 From: himars@REDACTED (Jack Tang) Date: Sun, 15 Jan 2017 14:09:41 +0800 Subject: [erlang-questions] How can I release beam process memory? In-Reply-To: References: Message-ID: Hi Daniel Thank you for the clue and excellent recon toolkits! I would like dig it deeper :) BR -Jack On Sun, Jan 15, 2017 at 4:47 AM, D?niel Szoboszlay wrote: > Hi Jack, > > I guess the 9 GB is lost due to memory fragmentation. Erlang allocates > memory in large chunks called carriers from the OS, then places the blocks > your program actually needs on these carriers. A carrier can only be > returned to the OS once all the blocks on it have been freed (and even > then, the memory allocator may decide to keep it around for a while in case > more memory is needed). > > You can check with recon_alloc > how much unused memory is > lost due to fragmentation in the various allocators. > > The bad news is that you cannot defragment the carriers, and if the > selected memory allocator strategy doesn't work well for your application, > you cannot change it either without restarting the emulator. > > However, if the memory is wasted in the eheap_alloc, you may try to force > a GC on all processes a couple of times. As the GC copies the memory, it > will allocate new blocks and free up the old heap blocks. So there's a > chance the allocators can compact the blocks together on fewer segments. > But that's just a guess, it may or may not work at all. > > Cheers, > Daniel > > On Sat, 14 Jan 2017 at 08:04 Jack Tang wrote: > >> Hello list, >> >> I run one Erlang application on Debian server and today I find the beam >> process consumes around 35G memory by `top` command. >> >> ``` >> KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers >> KiB Swap: 0 total, 0 used, 0 free. 18684864 cached >> Mem >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> 11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp >> ``` >> >> I connect to the Erlang application using remote shell and find the >> mem-leaked supervisor tree and run gc on the whole tree. Code looks like >> blow: >> >> ``` >> lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) >> end, supervisor:which_children(some_thing_sup)). >> ``` >> >> and erlang:memory() decreases from 32G to 23G. >> ``` >> [{total,22982011544}, >> {processes,12884182336}, >> {processes_used,12884170336}, >> {system,10097829208}, >> {atom,13828705}, >> {atom_used,13796692}, >> {binary,170530288}, >> {code,16450626}, >> {ets,9637717576}] >> ``` >> >> However, when I input `top` command, the beam process still takes 35G >> memory. What can I do to release the 9G memory? Thanks >> >> BR >> -Jack >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- Jack Tang http://www.linkedin.com/in/jacktang -------------- next part -------------- An HTML attachment was scrubbed... URL: From dgud@REDACTED Sun Jan 15 10:01:42 2017 From: dgud@REDACTED (Dan Gudmundsson) Date: Sun, 15 Jan 2017 09:01:42 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <587ADD95.6040109@gmail.com> References: <587ADD95.6040109@gmail.com> Message-ID: We have started to work on a new string module, we will make a PR when we have decided directions of how the api should look like. Basic stuff like uppercase, lowercase, to_nfc, to _nfd and gc (grapheme_clusters) are implemented for unicode:chardata() input, I have used elixir's module as inspiration. That is the easy part, writing a nice api on top of that is the hard part and naming the module something different than string. /Dan On Sun, Jan 15, 2017 at 3:25 AM Michael Truog wrote: > This thread started to talk about the need for unicode functionality in > Erlang and how it exists currently in Elixir but not in Erlang. I created > a repository with the Elixir functions created as Erlang functions in an > Erlang module as an example of what I want at > https://github.com/okeuday/unicode_data/ > > The generated module (unicode_data) only includes functions from the first > Erlang module contained in unicode.ex (Elixir.String.Unicode) though it > does bring up some important topics: > 1) Add the unicode version to the Erlang module version. The generated > unicode_data module has a timestamp suffix, so we know both the unicode > version and the timestamp when the unicode_data module was generated. > 2) Use only lists, not binaries, to make sure all temporary data stays on > the heap of the Erlang process. That should be best for performance, > though I haven't proved that with any performance testing. > > I haven't added tests, though I have compared the unicode_data Erlang > module, to the Elixir.String.Unicode module and the data looks correct. > Mainly thought this would help the discussion. > > Best Regards, > Michael > > > On 01/10/2017 10:58 AM, Bhag Chandra wrote: > > Hello, > > I have been coding in Erlang for 2 years. A wonderful language but not > very big community, so I cant discuss my questions with programmers around > me (Java, Python guys). I found out about this list today. > > I have some fundamental doubts about the Erlang. It would be great if > someone can help me clarify them. > > > 1) "Strings in Erlang are internally treated as a list of integers of each > character's ASCII values, this representation of string makes operations > faster. For example, string concatenation is constant time operation in > Erlang." Can someone explain why? > > 2) "It makes sense to use Erlang only where system's availability is very > high". Is it not a very general requirement of most of the systems? > Whatsapp to Google to FB to Amazon to Paypal to Barclays etc they all are > high availability systems, so we can use Erlang in all of them? > > 3) "Every message which is sent to a process, goes to the mailbox of that > process. When process is free, it consumes that message from mailbox". So > how exactly does process ask from the mailbox for that message? Is there a > mechanism in a process' memory which keeps polling its mailbox. I basically > want to understand how message is sent from mailbox to my code in process. > > 4) We say that a message is passed from process A to process B by simply > using a bang (!) character, but what happens behind the scenes to pass this > message? Do both processes establish a tcp connection first and then pass > message or what? > > 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) Mr. > Armstrong is talking about the difference between the context switching > overhead between OS threads and Erlang processes. He says, thread context > switching is of order 700 words but Erlang process context switching is ... > ? > I cant understand what he said, if someone could tell. > > > P.S. Please excuse for any grammatical errors, English is not my first > language. > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Jan 15 10:48:19 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Sun, 15 Jan 2017 10:48:19 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <587ADD95.6040109@gmail.com> Message-ID: Great work! Does this include (some) support for locales? As far as I recall they are necessary to do uppercasing and lowercasing properly. If this includes support for locales, or would in the future, may I suggest 'text' for the module name? Otherwise something else. :-) On 01/15/2017 10:01 AM, Dan Gudmundsson wrote: > We have started to work on a new string module, we will make a PR when > we have decided directions of how the api should look like. > > Basic stuff like uppercase, lowercase, to_nfc, to _nfd and gc > (grapheme_clusters) are implemented for unicode:chardata() input, I have > used elixir's module as inspiration. > > That is the easy part, writing a nice api on top of that is the hard > part and naming the module > something different than string. > > /Dan > > On Sun, Jan 15, 2017 at 3:25 AM Michael Truog > wrote: > > ||This thread started to talk about the need for unicode > functionality in Erlang and how it exists currently in Elixir but > not in Erlang. I created a repository with the Elixir functions > created as Erlang functions in an Erlang module as an example of > what I want at https://github.com/okeuday/unicode_data/ > > The generated module (unicode_data) only includes functions from the > first Erlang module contained in unicode.ex (Elixir.String.Unicode) > though it does bring up some important topics: > 1) Add the unicode version to the Erlang module version. The > generated unicode_data module has a timestamp suffix, so we know > both the unicode version and the timestamp when the unicode_data > module was generated. > 2) Use only lists, not binaries, to make sure all temporary data > stays on the heap of the Erlang process. That should be best for > performance, though I haven't proved that with any performance testing. > > I haven't added tests, though I have compared the unicode_data > Erlang module, to the Elixir.String.Unicode module and the data > looks correct. Mainly thought this would help the discussion. > > Best Regards, > Michael > > > On 01/10/2017 10:58 AM, Bhag Chandra wrote: >> Hello, >> >> I have been coding in Erlang for 2 years. A wonderful language >> but not very big community, so I cant discuss my questions with >> programmers around me (Java, Python guys). I found out about this >> list today. >> >> I have some fundamental doubts about the Erlang. It would be great >> if someone can help me clarify them. >> >> >> 1) "Strings in Erlang are internally treated as a list of integers >> of each character's ASCII values, this representation of string >> makes operations faster. For example, string concatenation is >> constant time operation in Erlang." Can someone explain why? >> >> 2) "It makes sense to use Erlang only where system's availability >> is very high". Is it not a very general requirement of most of >> the systems? Whatsapp to Google to FB to Amazon to Paypal to >> Barclays etc they all are high availability systems, so we can use >> Erlang in all of them? >> >> 3) "Every message which is sent to a process, goes to the mailbox >> of that process. When process is free, it consumes that message >> from mailbox". So how exactly does process ask from the mailbox >> for that message? Is there a mechanism in a process' memory which >> keeps polling its mailbox. I basically want to understand how >> message is sent from mailbox to my code in process. >> >> 4) We say that a message is passed from process A to process B by >> simply using a bang (!) character, but what happens behind the >> scenes to pass this message? Do both processes establish a tcp >> connection first and then pass message or what? >> >> 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) >> Mr. Armstrong is talking about the difference between the context >> switching overhead between OS threads and Erlang processes. He >> says, thread context switching is of order 700 words but Erlang >> process context switching is ... ? >> I cant understand what he said, if someone could tell. >> >> >> P.S. Please excuse for any grammatical errors, English is not my >> first language. >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From mikpelinux@REDACTED Sun Jan 15 10:58:26 2017 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Sun, 15 Jan 2017 10:58:26 +0100 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> Message-ID: <22651.18370.454189.186497@gargle.gargle.HOWL> REIX, Tony writes: > > Hi, > > I'd like to port Erlang/OTP on AIX. > > After browsing several files and ReadMe files in the source and on Erlang web-site, it seems to me that AIX is not yet handled by Erlang. > > I've already started to look at the code. And it appears that there is some assembler code for PowerPC, probably for Linux. > > However, browsing the code, I see that it talks of ELF (AIX is XCOFF) and of AIX. I'm quite lost. > Moreover, I've read that only 32bit Linux/PowerPC is handled. No 64bits. Correct ? > > I'd like to find information about: > - which parts must be modified/extended for handling AIX (XCOFF) ? > - which parts must be modified/extended for handling 64bit in addition to PowerPC 32bit ? The Erlang/OTP VM should build without using any of that assembly code. What you need is: - a POSIX environment (see HOWTO/INSTALL.md) - a decent ANSI-C or better yet ISO-C compiler, GCC is preferred - a libc with pthreads (even that might be optional, c.f. Windows, but might need configure-time overrides) - a termcap or ncurses library (optional but useful for the Erlang shell; --without-termcap if you don't have it) Note that the VM builds for Windows which is PE-COFF, so there is no hard dependency on ELF. HiPE (the native code stuff) does work on 64-bit PowerPC, but the runtime support is written for Linux (ELF) and some older MacOS so would need updating for XCOFF and the AIX assembler, and re-validation for however the AIX ABI looks these days. Just pass --disable-hipe when configuring and ignore that for now. Threading uses ordinary pthreads. --disable-threads if the AIX threads are too different. Threading and SMP support wants atomics. That's a bit of a mess, using GCC inline assembly code and fallbacks using (I think) pthreads synchronization primitives and/or libatomic. Either way, that code should build and work without any use of inline assembly code. Try --disable-threads --disable-smp-support for an initial, basic, but functioning VM. From dgud@REDACTED Sun Jan 15 11:17:17 2017 From: dgud@REDACTED (Dan Gudmundsson) Date: Sun, 15 Jan 2017 10:17:17 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <587ADD95.6040109@gmail.com> Message-ID: On Sun, Jan 15, 2017 at 10:48 AM Lo?c Hoguin wrote: > Great work! > You have not seen it yet :-) > > Does this include (some) support for locales? As far as I recall they > are necessary to do uppercasing and lowercasing properly. > > No not currently I have taken the elixir approach of a basic unicode support without locale handling, that is another can of worm which I guess can be opened later if there is need. But we don't have locale support anywhere else and I don't want start with adding that first, then will we never get any unicode support. >From my understanding for uppercase and lowercase there very few cases which need the locale to correctly transform them. /Dan > If this includes support for locales, or would in the future, may I > suggest 'text' for the module name? Otherwise something else. :-) > > On 01/15/2017 10:01 AM, Dan Gudmundsson wrote: > > We have started to work on a new string module, we will make a PR when > > we have decided directions of how the api should look like. > > > > Basic stuff like uppercase, lowercase, to_nfc, to _nfd and gc > > (grapheme_clusters) are implemented for unicode:chardata() input, I have > > used elixir's module as inspiration. > > > > That is the easy part, writing a nice api on top of that is the hard > > part and naming the module > > something different than string. > > > > /Dan > > > > On Sun, Jan 15, 2017 at 3:25 AM Michael Truog > > wrote: > > > > ||This thread started to talk about the need for unicode > > functionality in Erlang and how it exists currently in Elixir but > > not in Erlang. I created a repository with the Elixir functions > > created as Erlang functions in an Erlang module as an example of > > what I want at https://github.com/okeuday/unicode_data/ > > > > The generated module (unicode_data) only includes functions from the > > first Erlang module contained in unicode.ex (Elixir.String.Unicode) > > though it does bring up some important topics: > > 1) Add the unicode version to the Erlang module version. The > > generated unicode_data module has a timestamp suffix, so we know > > both the unicode version and the timestamp when the unicode_data > > module was generated. > > 2) Use only lists, not binaries, to make sure all temporary data > > stays on the heap of the Erlang process. That should be best for > > performance, though I haven't proved that with any performance > testing. > > > > I haven't added tests, though I have compared the unicode_data > > Erlang module, to the Elixir.String.Unicode module and the data > > looks correct. Mainly thought this would help the discussion. > > > > Best Regards, > > Michael > > > > > > On 01/10/2017 10:58 AM, Bhag Chandra wrote: > >> Hello, > >> > >> I have been coding in Erlang for 2 years. A wonderful language > >> but not very big community, so I cant discuss my questions with > >> programmers around me (Java, Python guys). I found out about this > >> list today. > >> > >> I have some fundamental doubts about the Erlang. It would be great > >> if someone can help me clarify them. > >> > >> > >> 1) "Strings in Erlang are internally treated as a list of integers > >> of each character's ASCII values, this representation of string > >> makes operations faster. For example, string concatenation is > >> constant time operation in Erlang." Can someone explain why? > >> > >> 2) "It makes sense to use Erlang only where system's availability > >> is very high". Is it not a very general requirement of most of > >> the systems? Whatsapp to Google to FB to Amazon to Paypal to > >> Barclays etc they all are high availability systems, so we can use > >> Erlang in all of them? > >> > >> 3) "Every message which is sent to a process, goes to the mailbox > >> of that process. When process is free, it consumes that message > >> from mailbox". So how exactly does process ask from the mailbox > >> for that message? Is there a mechanism in a process' memory which > >> keeps polling its mailbox. I basically want to understand how > >> message is sent from mailbox to my code in process. > >> > >> 4) We say that a message is passed from process A to process B by > >> simply using a bang (!) character, but what happens behind the > >> scenes to pass this message? Do both processes establish a tcp > >> connection first and then pass message or what? > >> > >> 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) > >> Mr. Armstrong is talking about the difference between the context > >> switching overhead between OS threads and Erlang processes. He > >> says, thread context switching is of order 700 words but Erlang > >> process context switching is ... ? > >> I cant understand what he said, if someone could tell. > >> > >> > >> P.S. Please excuse for any grammatical errors, English is not my > >> first language. > >> > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > https://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Sun Jan 15 18:26:22 2017 From: lukas@REDACTED (Lukas Larsson) Date: Sun, 15 Jan 2017 18:26:22 +0100 Subject: [erlang-questions] How can I release beam process memory? In-Reply-To: References: Message-ID: Hello Frank, On Sun, Jan 15, 2017 at 12:03 AM, Frank Muller wrote: > I would appreciate if the OTP team can teach us how to debug VM's mem > issues, and how pick the right allocator. > The right allocator is the default allocator. It is only in very rare circumstances that you would want to change it. Sure you can optimize them to be more perfect, but in most cases the defaults are great. If you suspect that you have issues that tuning erts_alloc will solve the best starting points is to read and understand Fred Hebert's excellent blogs/manuals about using erts_alloc to combat memory fragmentation. I cannot stress enough how great a resource his blog post ( https://blog.heroku.com/logplex-down-the-rabbit-hole) and manual ( https://www.erlang-in-anger.com/) are. Then you may also want to watch mine and Rickard Green's/Patrik Nyblom's talks on the subject, they basically cover the same things that Fred covers, but some parts in greater detail. There is also some internal documentation in https://github.com/erlang/otp/tree/master/erts/emulator/internal_doc that explain some of the features that you can tune in erts_alloc. There are also quite a few mail conversations on this mailing list about these types of problems. When it comes to allocator tuning there is no magic formula. We attempted to build something that automatically tuned the settings in the instrument module (http://erlang.org/doc/man/instrument.html), but it didn't turn out very useful. The problems that to pop up with erts_alloc seem to be very different in both symptoms and solutions. If you explain what it is that you have issues with (backing the claims with recon_alloc statistics) and what solutions you have tried then we'll very likely answer. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From andras.boroska@REDACTED Sun Jan 15 19:43:49 2017 From: andras.boroska@REDACTED (=?UTF-8?Q?Boroska_Andr=C3=A1s?=) Date: Sun, 15 Jan 2017 19:43:49 +0100 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <22651.18370.454189.186497@gargle.gargle.HOWL> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> Message-ID: Hi Tony, On Friday I was too quick to reply and among others I missed the part that you are from Bull. :) I used your R1502 version as the starting point. Still, let me add a few more hints. Mind you my memory can fail (I was working with AIX ~ 2 years ago). Hopefully the experts will correct me if I got something wrong. View it as my 2 cents. It is not comprehensive as you will realise it soon. > ?what are the main features that must be made available ?? AIX users tend to be large banks and similar organisations, and I imagine many of them interested in running RabbitMQ. So, based on my limited knowledge on the matter if you can start RabbitMQ that might be good enough for many. To list the minimal features I would say that SMP, 64bit, SSL are very likely. Others readers might add theirs too. HIPE: Hipe is mainly a performance boost for some applications, but many Erlang use cases do not require it and as Mikael said you can ignore it for now. Linux is always there for people preferring performance over AIX features. :) SMP: On the other hand I would argue that enabling smp support is probably a must for most use cases and it is not too difficult to achieve. One of the issues is that the AIX provided Assembler does not support the local labels syntax of the GNU Assembler. So, one either experiments with the GNU Assembler or change the labels and jumps in erts/include/internal/ppc32/atomic.h. See [1] and [2]. One can replace the label ?1:\t? with ?L1%=:\t? and replace the corresponding jump ?bne- 1b\n\t? with ?bne- L1%=\n\t?. Of course use a different label each time (E.g. L1, L2, L3, ...) This might or might not have performance implications, but definitely better than not having SMP support. Since Power8 offers up to 96 simultaneous threads, scheduler binding might be of interest of some potential users. To achieve that replace [3] and [4] with: #include ?sys/thread.h? bindprocessor(BINDTHREAD, thread_self(), (cpu_t) cpu) or something similar. 64 bit and compiler/linker flags: OBJECT_MODE=64 env variable needs to be exported obviously. GCC needs the following flags to link the executable: -Wl,-bexpall,-brtl,-b64 -maix64 For example you might have success bootstrapping by: CFLAGS=?-maix64 -g -O2 -D_LINUX_SOURCE_COMPAT -L/opt/freeware/lib64 -L/opt/freeware/lib? \ LDFLAGS=?-Wl,-bexpall,-brtl,-b64,-blibpath:/opt/freeware/lib64:opt/freeware/lib -maix64? \ AR=?ar -X64? \ ./otp_build autoconf --with-ssl=/opt/freeware --build= --disable-hipe and then the same flags for running ./otp_build configure Optionally you can also put the flags into the respective conf files. :) Maybe some apps will not compile, like os_mon or odbc. Tip to proceed quickly: you can skip any app by placing an empty file called SKIP in their directory. (lib/*/SKIP) OTHER: Note that the C standard allows the malloc(0) system call to return either zero or a pointer that cannot be dereferenced. Unixes typically return a non-zero pointer, but AIX returns 0 by default. The C interface of Erlang (erl_interface) calls malloc(0) for empty strings and checks the return value against 0 to catch allocation errors. To prevent core dumps in such cases it helps to define -D_LINUX_SOURCE_COMPAT which enforces the non-zero pointer returns. Defining it will probably not hurt and users of C drivers will appreciate. As I mentioned R18 reworked clock/time handling. Expect some incompatibilities in that area. The missing vsyslog on AIX can potentially be worked around by using printf. Or perhaps you know a better solution. As openssl versions seems to go obsolete rather quickly, expect that many users will be interested not only in the binary but also in your source rpm package when you upload to BullFreeware, so they can potentially compile and link with a different openssl version. BASIC TESTING: To check if core/thread number detection works start the Erlang shell and note the smp information in the first line. It should match the number of CPU threads you have. For example [smp:8:8]. See [5] Also note the presence of [64-bit] on the same line. Bonus (may not be critical for many) if you can bind schedulers, for example: erl +sct L0-7c0-7 +stbt db +S8:8 then erlang:system_info(scheduler_bindings). returns {0,1,2,....} and erlang:system_info(cpu_topology). returns [{processor,[{core,{logical,0}},{core,{logical,1}},{core,{logical,2}}, ... }]}] While in the shell issue crypto:start(). that should return ok if linking with ssl worked. There are also test suites for the emulator. You can run ./otp_build setup with the above CFLAGS, AR, LDFLAGS, add your bin dir to the PATH and run ./otp_build tests. Change dir to release/tests/test_server, run erl, then ts:install(), ts:run(emulator, [batch]). Take the above information with a grain of salt as I have no access to AIX and not an expert on building Erlang. The OTP team or others might correct my mistakes. Thank you for your work! Best regards, Andras [1] https://gcc.gnu.org/ml/gcc/2005-12/msg00671.html [2] http://stackoverflow.com/questions/3898435/labels-in-gcc-inline-assembly [3] https://github.com/erlang/otp/blob/maint-19/erts/lib_src/common/erl_misc_utils.c#L591 [4] https://github.com/erlang/otp/blob/maint-19/erts/configure.in#L2294 [5] http://erlang.org/doc/man/erl.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickard.m.andersson@REDACTED Sun Jan 15 10:36:28 2017 From: rickard.m.andersson@REDACTED (Rickard Andersson) Date: Sun, 15 Jan 2017 11:36:28 +0200 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: <5b1428f2-895f-3b2b-586e-e78981e202b7@gmail.com> No one is talking about doing `"Here's a string".upcase()`, though. You seem to be under the impression that Elixir uses methods and objects or the like to accomplish reasonable string operations. It uses a String module that has functions. It's not different from Erlang, it just actually implements things that people want. // gonz On 14-Jan-17 22:12, Garrett Smith wrote: > On Sat, Jan 14, 2017 at 11:08 AM, John Doe wrote: >>> string:to_upper("hello"). >> This function works for ASCII only. That's the problem. MICHA? = >> string:to_upper("Micha?"). > Neither does the string module work on binaries, which I've found > inconvenient. Fortunately you can just use an alternative > function/module. > > Sometimes I have to write my own function, which is also inconvenient. > > Modifying built-in type behavior (the vaunted Ruby string, but also > Python, etc.) is super extremely inconvenient. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From michel.boaventura@REDACTED Mon Jan 16 04:29:34 2017 From: michel.boaventura@REDACTED (Michel Boaventura) Date: Mon, 16 Jan 2017 01:29:34 -0200 Subject: [erlang-questions] Problem building plt to dialyzer Message-ID: Hi, I've installed erlang 19.2 from source code on my Gentoo Linux and I'm not able to build plt to dialyzer. It gives me the above error. The build was configured with --without-javac --prefix=/usr dialyzer --build_plt --apps kernel dialyzer: Could not get abstract code for file: /usr/lib/erlang/lib/kernel-5.1.1/ebin/disk_log.beam (please recompile it with +debug_info) I've also tried entering on erts/emulator and running make debug and tried to set ERL_COMPILE_FLAGS="+debug_info" with no luck. I don't know if it is an issue with gentoo (both my gentoo machines have the same problem) because I have the same issue using portage and evm to install it. Thank you! -- Michel Almada de Castro Boaventura Analista de Sistemas Laborat?rio de Software Livre - LSL -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Jan 16 09:03:54 2017 From: ok@REDACTED (ok@REDACTED) Date: Mon, 16 Jan 2017 21:03:54 +1300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: <8b015e2eb53466ddcbb8ba17a5b01575.squirrel@chasm.otago.ac.nz> Has anyone pointed out that there already exists an Erlang binding for ICU? The ICU Faq links to related projects which links to icu4e: https://github.com/beerriot/icu4e I have never used this or even inspected it, I merely note that it ALREADY EXISTS and is worth considering. I looked at ICU4C years ago and recoiled in horror at the vastness of it. However, it does much more than just strings. If you know what the CLDR is (and if you do Unicode, who doesn't?) then you probably care that ICU handles that. If you do any calendar arithmetic, you probably care that ICU4C does calendar stuff (using CLDR) in addition to "just Unicode". (Of course if your timestamps are in UTC, it is logically *impossible* to compute timestamps correctly past December 31 this year -- we have been promised no leap second in June -- and it is somewhere between fiendishly hard and dubious to compute timestamps before 1 Jan 1972, but you already know that. I have no idea what ICU does about this issue.) From seriy.pr@REDACTED Mon Jan 16 11:52:49 2017 From: seriy.pr@REDACTED (=?UTF-8?B?0KHQtdGA0LPQtdC5INCf0YDQvtGF0L7RgNC+0LI=?=) Date: Mon, 16 Jan 2017 13:52:49 +0300 Subject: [erlang-questions] [ANN] Epgsql 3.3.0 Message-ID: Hi everybody! I'm glad to announce new epgsql release 3.3.0. https://github.com/epgsql/epgsql/releases/tag/3.3.0 Here is the list of changes: * Streaming replication protocol support (epgsql may act like PostgreSQL slave) https://github.com/epgsql/epgsql/blob/devel/streaming.md * Connection options now may be passed as map() * More error extra data fields returned in #error{extra} http://www.postgresql.org/docs/current/interactive/protocol-error-fields.html * Prefer non-localized severity in #error{severity} when available * Added set_notice_receiver/2 function to dynamically change receiver for asynchronous notifications * Asynchronous notifications receiver may be set to registered process name (atom) * get_cmd_status/1 function added** * Fixed return value of `DELETE ... RETURNING ..` in case when 0 rows were deleted (now returns `{ok, 0, Columns, []}` instead of `{ok, 0}`) * TCP socket implicitly closed when epgsql connection process terminates * Some typespecs fixed ** The main practical usecase is to detect if transaction was really committed after COMMIT. {ok, _, _} = epgsql:squery(C, "BEGIN"). {error, _} = epgsql:equery(C, "SELECT 1 / $1::integer", [0]). {ok, _, _} = epgsql:squery(C, "COMMIT"). {ok, rollback} = epgsql:get_cmd_status(C). Thanks to all contributors and reviewers: Alexander Sedov David Welton Ivan Dyachkov Mikhail Kalashnikov Sergey Prokhorov --- Epgsql team. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Mon Jan 16 12:06:24 2017 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 16 Jan 2017 12:06:24 +0100 Subject: [erlang-questions] SSL: Getting master_secret and client_random (or premaster_secret) In-Reply-To: <92895e0e-91df-2239-7eb7-bd068fab7759@gmail.com> References: <92895e0e-91df-2239-7eb7-bd068fab7759@gmail.com> Message-ID: 2017-01-13 21:27 GMT+01:00 Kenneth Lakin : > On 01/13/2017 10:57 AM, Ingela Andin wrote: > > Yes interop vs security can be a tradeoff. All these needs the user to > make > > an active choise. > > Notably, that active choice _doesn't_ include forcing the programmer to > also start the connection in "interop mode". Marking those ssl_options > with Big Red Dire Warnings was (correctly) deemed quite enough notice. :) > > Heck, verify_fun is documented as normal, non-hazardous (that is, it > lacks a Big Red Warning box) API, but can be misused to (accidentally) > seriously compromise one's connection security. verify_fun does _not_ > depend on an "enable_hazardous_options" connection flag. > > verify_fun can be abused (maybe we should add a warning) but it still in a smaller context however ..... > > This to ensure that it is the intent of the connection starter > > that this behaviour should be allowed. > > Given that the caller can bypass this check by pulling the PID out of > the SSLSocket, calling sys:get_state/1 and extracting the > security_parameters, this check seems to be more of an annoyance than > anything else. For instance, how would the author of the (hypothetical) > TLS connection pool library I'm using know that I _never_ will have a > legitimate need to extract the client_random from a connection it > establishes for me? > > > ... maybe I am being to paranoid ;) Well it seem like a good compromise would be that connection_information will only return these values when explicitly requested along with a warning in the documentation, as we can not protect against a malicious attacker that has access to the node anyway. Regards Ingela Erlang/OTP team - Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From garryh@REDACTED Mon Jan 16 15:57:13 2017 From: garryh@REDACTED (Garry Hodgson) Date: Mon, 16 Jan 2017 09:57:13 -0500 Subject: [erlang-questions] cowboy etags problems Message-ID: <5898b54f-a395-9166-d17d-cd6f2c339ede@att.com> I have an application that acts as an http proxy, accepting REST api calls, forwarding to appropriate endpoints, and returning result. It's built atop Cowboy, and generally works well. However, while exploring a reported bug I discovered that I was not handling etags properly. Whenever an if-match header was included, Cowboy returned a 400, before my code ever saw the request. I eventually discovered that I was being sent headers like: If-Match: 1234567 Instead of: If-Match: "1234567" Digging into the RFC it appears that Cowboy is handling this properly, as the surrounding quotes are part of the spec. But it will still cause problems for my users. When I pass the correct If-Match header, I get a 412 Precondition Failed. This also appears to be correct behavior, as my Cowboy instance doesn't have the etag, the eventual endpoint should. So I think what I really need to do is to turn off If-Match handling in Cowboy and just pass through the headers I get, or possibly intercept the request before Cowboy looks at this and massage the inputs/outputs. Am I correct in this diagnosis? Is there any way to solve this? It's kind of a showstopper. Any insights appreciated. Thanks From jesper.louis.andersen@REDACTED Mon Jan 16 17:00:39 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 16 Jan 2017 16:00:39 +0000 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <587ADD95.6040109@gmail.com> Message-ID: I'd suggest looking a bit at what the Go people are doing here as well. They have this problem at a large scale at Google, so I expect them to have handling of some of the nastier corner cases. Some observations: * They keep part of the unicode handling outside of the standard library. This allows it to develop independently of the main code base, which in some cases is nice. The things that seems to be in the separate package are collation, normalization, and canocalization. I have a hunch this happens because these things tend to update a bit like time zones and SSL top-level certificates: haphazardly. By keeping it outside, you avoid the problem of having to wait for a new major release, or even having to push a new minor release because some random pacific island decided on their monthly meeting to switch their collation rules :) * Keep it rather simple for starters. The Elixir set is probably what "most people need supported". * Currently, I'm mostly interested in normalization and canocalization. I have some data where this would be highly useful to have access to directly from Erlang. On Sun, Jan 15, 2017 at 11:17 AM Dan Gudmundsson wrote: > On Sun, Jan 15, 2017 at 10:48 AM Lo?c Hoguin wrote: > > Great work! > > > You have not seen it yet :-) > > > > Does this include (some) support for locales? As far as I recall they > are necessary to do uppercasing and lowercasing properly. > > > No not currently I have taken the elixir approach of a basic unicode > support without locale handling, that is another can of worm which I guess > can be opened later if there is need. > But we don't have locale support anywhere else and I don't want start with > adding that first, > then will we never get any unicode support. > From my understanding for uppercase and lowercase there very few cases > which need > the locale to correctly transform them. > > /Dan > > > If this includes support for locales, or would in the future, may I > suggest 'text' for the module name? Otherwise something else. :-) > > On 01/15/2017 10:01 AM, Dan Gudmundsson wrote: > > We have started to work on a new string module, we will make a PR when > > we have decided directions of how the api should look like. > > > > Basic stuff like uppercase, lowercase, to_nfc, to _nfd and gc > > (grapheme_clusters) are implemented for unicode:chardata() input, I have > > used elixir's module as inspiration. > > > > That is the easy part, writing a nice api on top of that is the hard > > part and naming the module > > something different than string. > > > > /Dan > > > > On Sun, Jan 15, 2017 at 3:25 AM Michael Truog > > wrote: > > > > ||This thread started to talk about the need for unicode > > functionality in Erlang and how it exists currently in Elixir but > > not in Erlang. I created a repository with the Elixir functions > > created as Erlang functions in an Erlang module as an example of > > what I want at https://github.com/okeuday/unicode_data/ > > > > The generated module (unicode_data) only includes functions from the > > first Erlang module contained in unicode.ex (Elixir.String.Unicode) > > though it does bring up some important topics: > > 1) Add the unicode version to the Erlang module version. The > > generated unicode_data module has a timestamp suffix, so we know > > both the unicode version and the timestamp when the unicode_data > > module was generated. > > 2) Use only lists, not binaries, to make sure all temporary data > > stays on the heap of the Erlang process. That should be best for > > performance, though I haven't proved that with any performance > testing. > > > > I haven't added tests, though I have compared the unicode_data > > Erlang module, to the Elixir.String.Unicode module and the data > > looks correct. Mainly thought this would help the discussion. > > > > Best Regards, > > Michael > > > > > > On 01/10/2017 10:58 AM, Bhag Chandra wrote: > >> Hello, > >> > >> I have been coding in Erlang for 2 years. A wonderful language > >> but not very big community, so I cant discuss my questions with > >> programmers around me (Java, Python guys). I found out about this > >> list today. > >> > >> I have some fundamental doubts about the Erlang. It would be great > >> if someone can help me clarify them. > >> > >> > >> 1) "Strings in Erlang are internally treated as a list of integers > >> of each character's ASCII values, this representation of string > >> makes operations faster. For example, string concatenation is > >> constant time operation in Erlang." Can someone explain why? > >> > >> 2) "It makes sense to use Erlang only where system's availability > >> is very high". Is it not a very general requirement of most of > >> the systems? Whatsapp to Google to FB to Amazon to Paypal to > >> Barclays etc they all are high availability systems, so we can use > >> Erlang in all of them? > >> > >> 3) "Every message which is sent to a process, goes to the mailbox > >> of that process. When process is free, it consumes that message > >> from mailbox". So how exactly does process ask from the mailbox > >> for that message? Is there a mechanism in a process' memory which > >> keeps polling its mailbox. I basically want to understand how > >> message is sent from mailbox to my code in process. > >> > >> 4) We say that a message is passed from process A to process B by > >> simply using a bang (!) character, but what happens behind the > >> scenes to pass this message? Do both processes establish a tcp > >> connection first and then pass message or what? > >> > >> 5) At 30:25 in this video ( https://youtu.be/YaUPdgtUYko?t=1825 ) > >> Mr. Armstrong is talking about the difference between the context > >> switching overhead between OS threads and Erlang processes. He > >> says, thread context switching is of order 700 words but Erlang > >> process context switching is ... ? > >> I cant understand what he said, if someone could tell. > >> > >> > >> P.S. Please excuse for any grammatical errors, English is not my > >> first language. > >> > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > https://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.reix@REDACTED Mon Jan 16 17:12:26 2017 From: tony.reix@REDACTED (REIX, Tony) Date: Mon, 16 Jan 2017 16:12:26 +0000 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <22651.18370.454189.186497@gargle.gargle.HOWL> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> Message-ID: <3c68e5c3-b6ec-da17-2e2a-aacacff1b037@atos.net> Hello Mikael, Thanks for the information ! I'm using GCC v4.8.4 . In general, porting an OpenSource package on AIX is only a matter of small and very small issues, except when assembler is used (Ruby, GCC, GCC Go, Erlang, ...). After building with Hipe, I've tried without Hipe. And I got the same result: an infinite loop, in read() from signal_dispatcher_thread_func(). In: ERLC ../ebin/hipe_rtl.beam with Hipe (--enable-hipe) In: ERLC ../ebin/hipe_consttab.beam without Hipe (--disable-hipe) . No idea why "hipe" appears here also. So, for now, I have a common issue, with or without hipe. Full configure is (without hipe): ./configure ppc-ibm-aix6.1 --prefix=/opt/freeware --exec-prefix=/opt/freeware --bindir=/opt/freeware/bin --sbindir=/opt/freeware/sbin --sysconfdir=/opt/freeware/etc --datadir=/opt/freeware/share --includedir=/opt/freeware/include --libdir=/opt/freeware/lib --libexecdir=/opt/freeware/libexec --localstatedir=/opt/freeware/var --sharedstatedir=/opt/freeware/com --mandir=/opt/freeware/man --infodir=/opt/freeware/info --enable-shared-zlib --enable-sctp --disable-systemd --disable-hipe --without-jinterface --without-common_test --without-debugger --without-dialyzer --without-et --without-megaco --without-observer --without-reltool --without-typer --without-wx configure: WARNING: you should use --build, --host, --target Investigating. About ELF/XCOFF, thanks for informing me that this part should not be too big. Now, when building with No SMP support in addition to No threads and No Hipe, the build goes further, up tow Java building: JAVAC /opt/freeware/src/packages/BUILD/otp_src_19.2/lib/ic/priv/com/ericsson/otp/ic/Environment.class Environment.java:41: package com.ericsson.otp.erlang does not exist private com.ericsson.otp.erlang.OtpSelf self; ^ I'll investigate later. Now, building with : --disable-hipe --disable-threads --disable-smp-support --without-javac I have an issue with LD: LD ../priv/lib/powerpc-ibm-aix6.1/dyntrace.so ld: 0706-012 The -h flag is not recognized. ld: 0706-012 The -a flag is not recognized. I'll investigate, fix and go further. Thanks for indicating the main options of ./configure of Erlang to study at first. Regards Tony Regards, Tony Le 15/01/2017 ? 10:58, Mikael Pettersson a ?crit : REIX, Tony writes: > > Hi, > > I'd like to port Erlang/OTP on AIX. > > After browsing several files and ReadMe files in the source and on Erlang web-site, it seems to me that AIX is not yet handled by Erlang. > > I've already started to look at the code. And it appears that there is some assembler code for PowerPC, probably for Linux. > > However, browsing the code, I see that it talks of ELF (AIX is XCOFF) and of AIX. I'm quite lost. > Moreover, I've read that only 32bit Linux/PowerPC is handled. No 64bits. Correct ? > > I'd like to find information about: > - which parts must be modified/extended for handling AIX (XCOFF) ? > - which parts must be modified/extended for handling 64bit in addition to PowerPC 32bit ? The Erlang/OTP VM should build without using any of that assembly code. What you need is: - a POSIX environment (see HOWTO/INSTALL.md) - a decent ANSI-C or better yet ISO-C compiler, GCC is preferred - a libc with pthreads (even that might be optional, c.f. Windows, but might need configure-time overrides) - a termcap or ncurses library (optional but useful for the Erlang shell; --without-termcap if you don't have it) Note that the VM builds for Windows which is PE-COFF, so there is no hard dependency on ELF. HiPE (the native code stuff) does work on 64-bit PowerPC, but the runtime support is written for Linux (ELF) and some older MacOS so would need updating for XCOFF and the AIX assembler, and re-validation for however the AIX ABI looks these days. Just pass --disable-hipe when configuring and ignore that for now. Threading uses ordinary pthreads. --disable-threads if the AIX threads are too different. Threading and SMP support wants atomics. That's a bit of a mess, using GCC inline assembly code and fallbacks using (I think) pthreads synchronization primitives and/or libatomic. Either way, that code should build and work without any use of inline assembly code. Try --disable-threads --disable-smp-support for an initial, basic, but functioning VM. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zzantozz@REDACTED Mon Jan 16 20:08:13 2017 From: zzantozz@REDACTED (Ryan Stewart) Date: Mon, 16 Jan 2017 19:08:13 +0000 Subject: [erlang-questions] Erlang 19 callbacks aren't attributes? Message-ID: There was a question about this on the list a few months back, but it was unanswered. It seems that in Erlang 19, a module's callbacks no longer appear in the module_info() as 'attributes'. They are, however, included in the abstract code (via beam_lib). I'm wondering if this was an intentional change or an oversight. Are callbacks supposed to show up under a module's module_info attributes like they used to? Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Mon Jan 16 20:39:25 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 16 Jan 2017 20:39:25 +0100 Subject: [erlang-questions] Patch package OTP 19.2.1 released Message-ID: <9f878557-89e4-0d1c-39ae-b81b0f5bd8a7@ericsson.com> Patch Package: OTP 19.2.1 Git Tag: OTP-19.2.1 Date: 2017-01-16 Trouble Report Id: OTP-14144 Seq num: seq13242 System: OTP Release: 19 Application: erts-8.2.1 Predecessor: OTP 19.2 Check out the git tag OTP-19.2.1, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- erts-8.2.1 ------------------------------------------------------ --------------------------------------------------------------------- Note! The erts-8.2.1 application can *not* be applied independently of other applications on an arbitrary OTP 19 installation. On a full OTP 19 installation, also the following runtime dependency has to be satisfied: -- sasl-3.0.1 (first satisfied in OTP 19.1) --- Fixed Bugs and Malfunctions --- OTP-14144 Application(s): erts Related Id(s): seq13242 Fix a quite rare bug causing VM crash during code loading and the use of export funs (fun M:F/A) of not yet loaded modules. Requires a very specfic timing of concurrent scheduler threads. Has been seen on ARM but can probably also occure on other architectures. Bug has existed since OTP R16. Full runtime dependencies of erts-8.2.1: kernel-5.0, sasl-3.0.1, stdlib-3.0 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- This patch package will only be available via GitHub: https://github.com/erlang/otp/tree/OTP-19.2.1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathaniel@REDACTED Mon Jan 16 20:54:11 2017 From: nathaniel@REDACTED (Nathaniel Waisbrot) Date: Mon, 16 Jan 2017 14:54:11 -0500 Subject: [erlang-questions] cowboy etags problems In-Reply-To: <5898b54f-a395-9166-d17d-cd6f2c339ede@att.com> References: <5898b54f-a395-9166-d17d-cd6f2c339ede@att.com> Message-ID: <1C133D70-D36F-437C-BAD2-3B3204138CB5@waisbrot.net> I agree that the quotes are surprising, but that's how that header works. Are you sure that you need to run with broken etags? I've used etags from Cowboy with all major browsers and it works correctly. If you've got a broken client you may want to consider fixing the client rather than breaking the server, because you'll just keep encountering surprises (for example, if your client is some home-brewed thing and then you start using a HTTP library that does etags correctly). But if you want the headers to work some other way, then it shouldn't be too hard. I'd just copy the Cowboy code and modify it as needed: https://github.com/ninenines/cowboy/blob/c5a9771ac4f7844b61b5e36bfc7eb371a082c560/src/cowboy_rest.erl#L673-L697 > On Jan 16, 2017, at 9:57 AM, Garry Hodgson wrote: > > I have an application that acts as an http proxy, accepting REST api calls, forwarding to appropriate endpoints, and returning result. It's built atop Cowboy, and generally works well. > > However, while exploring a reported bug I discovered that I was not handling etags properly. Whenever an if-match header was included, Cowboy returned a 400, before my code ever saw the request. I eventually discovered that I was being sent headers like: If-Match: 1234567 > > Instead of: If-Match: "1234567" > > Digging into the RFC it appears that Cowboy is handling this properly, as the surrounding quotes are part of the spec. But it will still cause problems for my users. > > When I pass the correct If-Match header, I get a 412 Precondition Failed. This also appears to be correct behavior, as my Cowboy instance doesn't have the etag, the eventual endpoint should. So I think what I really need to do is to turn off If-Match handling in Cowboy and just pass through the headers I get, or possibly intercept the request before Cowboy looks at this and massage the inputs/outputs. > > Am I correct in this diagnosis? Is there any way to solve this? It's kind of a showstopper. > > Any insights appreciated. > > Thanks > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From nathaniel@REDACTED Mon Jan 16 21:39:54 2017 From: nathaniel@REDACTED (Nathaniel Waisbrot) Date: Mon, 16 Jan 2017 15:39:54 -0500 Subject: [erlang-questions] cowboy etags problems In-Reply-To: <0FB9733E-1A7E-4369-AA8B-C586FFBAF9BB@att.com> References: <5898b54f-a395-9166-d17d-cd6f2c339ede@att.com> <1C133D70-D36F-437C-BAD2-3B3204138CB5@waisbrot.net> <0FB9733E-1A7E-4369-AA8B-C586FFBAF9BB@att.com> Message-ID: <810E875C-4CD3-4D33-82E8-B4C906D25398@waisbrot.net> Oh sorry, I misunderstood your question. So what you want is to pass along the etags from your upstream, right? What sort of work is your code doing? Are you using Cowboy's REST flow? If your proxy server does some request validation and then passes the request along, maybe you want to do all the work in resource_exists (including sending the response) and return `{stop, Req, State}` from the callback to skip the rest of the handlers. > On Jan 16, 2017, at 3:00 PM, HODGSON, GARRY wrote: > > Thanks for your reply. I have, indeed, now told them their etags are wrong, showed them the rfc. > It?s really more the if-match handling that?s a problem. > > > On 1/16/17, 2:54 PM, "Nathaniel Waisbrot" wrote: > > I agree that the quotes are surprising, but that's how that header works. > > Are you sure that you need to run with broken etags? I've used etags from Cowboy with all major browsers and it works correctly. If you've got a broken client you may want to consider fixing the client rather than breaking the server, because you'll just keep encountering surprises (for example, if your client is some home-brewed thing and then you start using a HTTP library that does etags correctly). > > But if you want the headers to work some other way, then it shouldn't be too hard. I'd just copy the Cowboy code and modify it as needed: https://github.com/ninenines/cowboy/blob/c5a9771ac4f7844b61b5e36bfc7eb371a082c560/src/cowboy_rest.erl#L673-L697 > > > >> On Jan 16, 2017, at 9:57 AM, Garry Hodgson wrote: >> >> I have an application that acts as an http proxy, accepting REST api calls, forwarding to appropriate endpoints, and returning result. It's built atop Cowboy, and generally works well. >> >> However, while exploring a reported bug I discovered that I was not handling etags properly. Whenever an if-match header was included, Cowboy returned a 400, before my code ever saw the request. I eventually discovered that I was being sent headers like: If-Match: 1234567 >> >> Instead of: If-Match: "1234567" >> >> Digging into the RFC it appears that Cowboy is handling this properly, as the surrounding quotes are part of the spec. But it will still cause problems for my users. >> >> When I pass the correct If-Match header, I get a 412 Precondition Failed. This also appears to be correct behavior, as my Cowboy instance doesn't have the etag, the eventual endpoint should. So I think what I really need to do is to turn off If-Match handling in Cowboy and just pass through the headers I get, or possibly intercept the request before Cowboy looks at this and massage the inputs/outputs. >> >> Am I correct in this diagnosis? Is there any way to solve this? It's kind of a showstopper. >> >> Any insights appreciated. >> >> Thanks >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > From kostis@REDACTED Mon Jan 16 22:25:20 2017 From: kostis@REDACTED (Kostis Sagonas) Date: Mon, 16 Jan 2017 22:25:20 +0100 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <3c68e5c3-b6ec-da17-2e2a-aacacff1b037@atos.net> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> <3c68e5c3-b6ec-da17-2e2a-aacacff1b037@atos.net> Message-ID: <9ee1a3ca-b0fe-e65d-6b5f-ae415c23eb52@cs.ntua.gr> On 01/16/2017 05:12 PM, REIX, Tony wrote: > > After building with Hipe, I've tried without Hipe. > And I got the same result: an infinite loop, in read() from > signal_dispatcher_thread_func(). > In: ERLC ../ebin/hipe_rtl.beam with Hipe (--enable-hipe) > In: ERLC ../ebin/hipe_consttab.beam without Hipe (--disable-hipe) > . No idea why "hipe" appears here also. > So, for now, I have a common issue, with or without hipe. I can provide some info regarding the behavior you are experiencing: - Certain modules of the 'hipe' application, though not all. are compiled independently of whether HiPE is enabled or not because they are used by 'dialyzer'. - It just so happens that 'hipe' is the first application that is compiled in the bootstrapping process, so the infinite loop you are getting is not related to HiPE but shows some deeper problem in the system you have built. Bottomline: what you are experiencing is not HiPE-related. Your system is broken. Kostis From essen@REDACTED Mon Jan 16 22:27:00 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 16 Jan 2017 22:27:00 +0100 Subject: [erlang-questions] cowboy etags problems In-Reply-To: <5898b54f-a395-9166-d17d-cd6f2c339ede@att.com> References: <5898b54f-a395-9166-d17d-cd6f2c339ede@att.com> Message-ID: Write a middleware that changes the header and add the quotes back, if you really really want to support this behavior. Note that in Cowboy 2 that answer will change to "write a stream handler" which is a kind of low level middleware in Cowboy 2, and will fix the invalid header much earlier, ensuring it works across more than cowboy_rest. On 01/16/2017 03:57 PM, Garry Hodgson wrote: > I have an application that acts as an http proxy, accepting REST api > calls, forwarding to appropriate endpoints, and returning result. It's > built atop Cowboy, and generally works well. > > However, while exploring a reported bug I discovered that I was not > handling etags properly. Whenever an if-match header was included, > Cowboy returned a 400, before my code ever saw the request. I eventually > discovered that I was being sent headers like: If-Match: 1234567 > > Instead of: If-Match: "1234567" > > Digging into the RFC it appears that Cowboy is handling this properly, > as the surrounding quotes are part of the spec. But it will still cause > problems for my users. > > When I pass the correct If-Match header, I get a 412 Precondition > Failed. This also appears to be correct behavior, as my Cowboy instance > doesn't have the etag, the eventual endpoint should. So I think what I > really need to do is to turn off If-Match handling in Cowboy and just > pass through the headers I get, or possibly intercept the request before > Cowboy looks at this and massage the inputs/outputs. > > Am I correct in this diagnosis? Is there any way to solve this? It's > kind of a showstopper. > > Any insights appreciated. > > Thanks > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Lo?c Hoguin https://ninenines.eu From michael.nisi@REDACTED Mon Jan 16 23:05:29 2017 From: michael.nisi@REDACTED (Michael Nisi) Date: Mon, 16 Jan 2017 23:05:29 +0100 Subject: [erlang-questions] cowboy etags problems In-Reply-To: References: <5898b54f-a395-9166-d17d-cd6f2c339ede@att.com> Message-ID: <197D8BA5-CEF3-4521-B7A5-55DDE4D346C8@gmail.com> Evaluating proxies myself at the moment, I'd be interested to hear what made you choose cowboy, over haproxy or nginx, as a proxy? Except that it's Erlang, of course, which is as good a reason as any. Michael From ok@REDACTED Tue Jan 17 07:20:18 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 17 Jan 2017 19:20:18 +1300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: On 14/01/17 1:20 AM, Jesper Louis Andersen wrote: > When I wrote my post, I was--probably incorrectly--assuming the older > notion of a "String" where the representation is either ASCII or > something like ISO-8859-15. In this case, a string coincides with a > stream of bytes. Just to be picky, an ASCII string is a sequence of 7-bit code points, not bytes. I have had the (dis)pleasure of working with an operating system where for some unfathomable reason all ASCII characters were coded as 128+official code point. The OS is dead and they *deserved* it. *And* in ASCII, not all sequences are legal. Technically, according to the ASCII specification, NUL and DEL are ok on the wire, but are not allowed inside a string. (You thought Kernighan and Ritchie came up with NUL termination all by themselves? Nope, it was official well before C: strings should not contain NUL or DEL.) But it doesn't end there. ASCII supported multibyte characters. Yes, really. Some of the characters included in ASCII make sense only when you realise that they were *supposed* to be positionable diacriticals. So for example ? = ' BS e ? = ^ BS o (that's why ^ is in ASCII) ? = , BS c ? = ~ BS n (that's why ~ is in ASCII) ? = " BS u (so there are three uses for ") This is for real. It's not something I made up. It's one of the things that CHANGED between ASCII and ISO Latin 1. ASCII *does* allow making composite characters by overstriking and Latin 1 *doesn't*. So even ASCII was a much more subtle thing that most people realise. Anglophone monoglots didn't need accents, so software developed in the Anglosphere tended to pretend that overstrikes didn't exist, until the pretence became a de facto reality. (Another thing the Unix designers have been slammed for is using LF as line terminator instead of CR+LF. But in fact "New Line" was one of the legal readings for LF in the original ASCII design. Oh yeah, you could use CR for overstrikes as well...) Anyone else remember using ASCII on TOPS-10? Five 7-bit characters per 36-bit word, so not every sequence of words could be represented as a sequence of ASCII characters. And the uses people found for the extra bit? Or the programming languages that let you specify the byte size for an input file from 1 to 36 bits? The Good Old Days are *now*. > Data needs parsing. A lot of data comes in as some kind of stringy > representation: UTF-8, byte array (binary), and so on. And some of it comes in fragments which need reassembly, and some of it comes encrypted and/or compressed. I am currently struggling with a class of 3rd year students who have the ingrained belief, resulting from years of abuse -3dw exposure to Java, that - it is a good idea to turn anything that should be the key of a hash table into a string - strings are cheap - building very long strings by repeated concatenation of small strings is a wonderful idea ... I'm trying to get across to them the idea that - there are data representations you use at the BOUNDARIES of your program for input/output - there are data representations you use in your program for PROCESSING - there are different design forces involved, so - good representations for one purpose are seldom good representations for the other purpose. For example, if you're going to stick with strings, if someone hands you a "UTF-8" string and expects you to give it back later, you should almost certainly give it back VERBATIM, absolutely unchanged, but if you want to process it, you probably want to convert your working copy to some canonical form (which might or might not be a string). From ok@REDACTED Tue Jan 17 07:38:55 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 17 Jan 2017 19:38:55 +1300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> Message-ID: <12f0db3b-989e-a8fa-0017-f0213b27e155@cs.otago.ac.nz> On 14/01/17 11:34 AM, Micha? Muska?a wrote: > I fully agree there are no languages that deal with strings perfectly. > That said there are those that are better at it and those that aren't so > good. A language, where I need to look for a library to upcase or > downcase my own name, fits into the second group in my book. Let's see, I use Fortran, C, occasionally C++ or Objective C, Python, Prolog, Erlang, Lisp, Scheme, R, Ada (when Apple haven't broken my gcc setup *again*), Smalltalk, Java, C#, SML, F#, sometimes a bit of JavaScript (yuck). In *all* of them, if there are operations to convert case, the operations are in fact *library* operations. In *all* of them, I have to look them up. Many of these languages were defined before Unicode was dreamed of, and to the extent that they do Unicode casing at all, do it wrong. (Case conversion can change the length of a string, for not very exotic text.) There is even a script in Unicode were for several editions of the Unicode standard we were told that there *were* two cases but one of them was obsolete so that while you *could* convert, you shouldn't, but then they said oops, they are actually two scripts, and the one we said should not be converted to "lower case" actually has *another* block of characters that really *are* lower case and it's OK to use those. (Was it Georgian? I forget.) Which of course brings up the points that - case conversion for Private Use Area characters may be important, but it is by definition not defined by Unicode - the Unicode database *changes* frequently; they try to keep as much stable as they can, but sometimes case mappings for existing characters *do* change, and of course characters that were not previously defined may need conversion. I had a colleague talking to me today who is a D evangelist. He just loves the language as a better C than C and not stonkingly bad like C++. Today he was whingeing that Unicode is up to version 9.something but D was still only up to Unicode version 5.1, so the language *purported* to handle Unicode, but for his purposes, didn't. He looked into upgrading it himself (he's serious about liking D), but the tools they used to convert the Unicode table files to whatever it is that D uses in the library weren't in the repository. (I understand that they are now.) Don't forget that Unicode has multiple clones of ASCII. There are for example several differently styled clones for use in mathematics -- oops, Unicode wasn't supposed to do that kind of thing, and it's not done for Hebrew and Greek letters, also used in mathematics -- but *those* characters shouldn't be case converted. For example, in Physics g and G are quite different things. Here's a thing I would like, and it seems to me the kind of thing that *should* be in language standards. I want to generate a number in the user's locale. How do I get the user's digits? Last time I looked there were over 50 copies of the digits 0..9. So this stuff is insanely complex and changes frequently. This is EXACTLY the kind of stuff that belongs in libraries, not language cores. Note that I am agreeing with the idea of *better* libraries for Erlang. Absolutely. But it's one of those thing where it's always going to be easier to do something else first: paint the house, get another PhD, raise a child, ... From michal@REDACTED Tue Jan 17 08:17:10 2017 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Tue, 17 Jan 2017 08:17:10 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <12f0db3b-989e-a8fa-0017-f0213b27e155@cs.otago.ac.nz> References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> <12f0db3b-989e-a8fa-0017-f0213b27e155@cs.otago.ac.nz> Message-ID: <8dc034b0-1aa4-4b1d-a5f2-a5256bf16f48@Spark> On 17 Jan 2017, 07:39 +0100, Richard A. O'Keefe , wrote: > > Let's see, I use Fortran, C, occasionally C++ or Objective C, > Python, Prolog, Erlang, Lisp, Scheme, R, Ada (when Apple haven't > broken my gcc setup *again*), Smalltalk, Java, C#, SML, F#, > sometimes a bit of JavaScript (yuck). In *all* of them, if > there are operations to convert case, the operations are in fact > *library* operations. In *all* of them, I have to look them up. > I just checked and all: Python 3, Ruby, Java, C# (so probably F# as well) and JavaScript properly uppercase the character "?" using the built-in functions, without any additional libraries. Micha?. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.reix@REDACTED Tue Jan 17 10:14:48 2017 From: tony.reix@REDACTED (REIX, Tony) Date: Tue, 17 Jan 2017 09:14:48 +0000 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <9ee1a3ca-b0fe-e65d-6b5f-ae415c23eb52@cs.ntua.gr> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> <3c68e5c3-b6ec-da17-2e2a-aacacff1b037@atos.net> <9ee1a3ca-b0fe-e65d-6b5f-ae415c23eb52@cs.ntua.gr> Message-ID: <4095e85e-ee99-54b5-15f9-eb8e299c0e2b@atos.net> Hi Kostis, Thanks for your help ! Yes. I do understand now that my current issues are not HiPE-related. Good that someone can confirm this hypothesis I had. I've found that 2 configure.in files have no knowledge of AIX > version 4, so that they use a default *) case. Patching. Re-Building... Making progress ! Step by step... Tony Le 16/01/2017 ? 22:25, Kostis Sagonas a ?crit : On 01/16/2017 05:12 PM, REIX, Tony wrote: After building with Hipe, I've tried without Hipe. And I got the same result: an infinite loop, in read() from signal_dispatcher_thread_func(). In: ERLC ../ebin/hipe_rtl.beam with Hipe (--enable-hipe) In: ERLC ../ebin/hipe_consttab.beam without Hipe (--disable-hipe) . No idea why "hipe" appears here also. So, for now, I have a common issue, with or without hipe. I can provide some info regarding the behavior you are experiencing: - Certain modules of the 'hipe' application, though not all. are compiled independently of whether HiPE is enabled or not because they are used by 'dialyzer'. - It just so happens that 'hipe' is the first application that is compiled in the bootstrapping process, so the infinite loop you are getting is not related to HiPE but shows some deeper problem in the system you have built. Bottomline: what you are experiencing is not HiPE-related. Your system is broken. Kostis -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.reix@REDACTED Tue Jan 17 14:18:06 2017 From: tony.reix@REDACTED (REIX, Tony) Date: Tue, 17 Jan 2017 13:18:06 +0000 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <623cfbe6-eb50-f3a3-c983-7840263f8131@atos.net> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> <3c68e5c3-b6ec-da17-2e2a-aacacff1b037@atos.net> <9ee1a3ca-b0fe-e65d-6b5f-ae415c23eb52@cs.ntua.gr> <623cfbe6-eb50-f3a3-c983-7840263f8131@atos.net> Message-ID: <5b691c56-5f57-c1cb-1115-a0d1563b20a4@atos.net> Resending without the attachment. Tony Le 17/01/2017 ? 11:52, Tony REIX a ?crit : Hi, With : --disable-hipe --disable-threads --disable-smp-support --without-javac I've been able to build Erlang and to launch the tests by: ct_run -suite ../compiler_test/andor_SUITE -case t_orelse However, it fails at beginning: "Failed to start CTH", and it talks about a file "see the CT Log for details"" that I have no idea where it is. I've only found at: otp_src_19.2/release/tests/test_server : ./ct_run.ct@REDACTED/tests.compiler_test.andor_SUITE.t_orelse.logs/run.2017-01-17_11.14.50/suite.log but it does not help a lot Any idea which document/file I should have a look at in order to understand the root cause of the failing testing ? Any idea about the root cause of the failure ? I've also tried with : erl -s ts install -s ts run all_tests -s init stop . But it crashes: init terminating in do_boot Crash dump Looking at: erl_crash.dump , I see: Current Process CP: 0x00000000 (invalid) erl_crash.dump file is attached to this email. REMOVED I guess that I have to learn a little bit ( ;) ) more about how Erlang works. See below for details. Thanks Tony 1) Launch tests : + cd /opt/freeware/src/packages/BUILD/otp_src_19.2/release/tests//test_server + /opt/freeware/src/packages/BUILD/otp_src_19.2/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse Common Test starting (cwd is /opt/freeware/src/packages/BUILD/otp_src_19.2/release/tests/test_server) Eshell V8.2 (abort with ^G) (ct@REDACTED)1> Common Test: Running make in test directories... (ct@REDACTED)1> Recompile: andor_SUITE (ct@REDACTED)1> andor_SUITE.erl:83: Warning: this clause cannot match because of different types/sizes (ct@REDACTED)1> andor_SUITE.erl:86: Warning: this clause cannot match because a previous clause always matches (ct@REDACTED)1> andor_SUITE.erl:87: Warning: this clause cannot match because a previous clause always matches (ct@REDACTED)1> andor_SUITE.erl:92: Warning: this clause cannot match because of different types/sizes (ct@REDACTED)1> andor_SUITE.erl:169: Warning: the guard for this clause evaluates to 'false' (ct@REDACTED)1> andor_SUITE.erl:170: Warning: the guard for this clause evaluates to 'false' (ct@REDACTED)1> andor_SUITE.erl:170: Warning: the result of the expression is ignored (suppress the warning by assigning the expression to the _ variable) (ct@REDACTED)1> andor_SUITE.erl:171: Warning: the guard for this clause evaluates to 'false' (ct@REDACTED)1> andor_SUITE.erl:172: Warning: the guard for this clause evaluates to 'false' (ct@REDACTED)1> andor_SUITE.erl:179: Warning: this expression will fail with a 'badarg' exception (ct@REDACTED)1> andor_SUITE.erl:194: Warning: this clause cannot match because a previous clause at line 194 always matches (ct@REDACTED)1> andor_SUITE.erl:195: Warning: the guard for this clause evaluates to 'false' (ct@REDACTED)1> andor_SUITE.erl:196: Warning: the guard for this clause evaluates to 'false' (ct@REDACTED)1> andor_SUITE.erl:197: Warning: the guard for this clause evaluates to 'false' (ct@REDACTED)1> andor_SUITE.erl:227: Warning: this clause cannot match because a previous clause at line 227 always matches (ct@REDACTED)1> andor_SUITE.erl:228: Warning: this clause cannot match because a previous clause at line 228 always matches (ct@REDACTED)1> andor_SUITE.erl:229: Warning: this clause cannot match because a previous clause at line 229 always matches (ct@REDACTED)1> andor_SUITE.erl:230: Warning: the guard for this clause evaluates to 'false' (ct@REDACTED)1> Recompile: test_lib (ct@REDACTED)1> CWD set to: "/opt/freeware/src/packages/BUILD/otp_src_19.2/release/tests/test_server/ct_run.ct@REDACTED" (ct@REDACTED)1> TEST INFO: 1 test(s), 1 case(s) in 1 suite(s) (ct@REDACTED)1> Testing tests.compiler_test.andor_SUITE.t_orelse: Starting test, 1 test cases (ct@REDACTED)1> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Error detected: "Failed to start CTH, see the CT Log for details" - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (ct@REDACTED)1> Testing tests.compiler_test.andor_SUITE.t_orelse: *** FAILED {andor_SUITE,init_per_suite} *** (ct@REDACTED)1> Testing tests.compiler_test.andor_SUITE.t_orelse: TEST COMPLETE, 0 ok, 0 failed, 1 skipped of 1 test cases 2) ./ct_run.ct@REDACTED/tests.compiler_test.andor_SUITE.t_orelse.logs/run.2017-01-17_11.14.50/suite.log === Suite started at 2017-01-17 11:14:50 === Starting test, 1 test cases =cases 1 =user root =host hardy1 =hosts hardy1 =emulator_vsn 8.2 =emulator beam =otp_release 19 =started 2017-01-17 11:14:50 =case andor_SUITE:init_per_suite =logfile andor_suite.init_per_suite.html =started 2017-01-17 11:14:50 =ended 2017-01-17 11:14:50 =result failed: "Failed to start CTH, see the CT Log for details", [{andor_SUITE, init_per_suite}] === *** FAILED {andor_SUITE,init_per_suite} *** === =case andor_SUITE:t_orelse =started 2017-01-17 11:14:50 =result auto_skipped: init_per_suite failed === *** Skipping test case #1 {andor_SUITE,t_orelse} *** === =case andor_SUITE:end_per_suite =started 2017-01-17 11:14:50 =result auto_skipped: init_per_suite failed === *** Skipping {andor_SUITE,end_per_suite} *** === TEST COMPLETE, 0 ok, 0 failed, 1 skipped of 1 test cases =finished 2017-01-17 11:14:50 =failed 0 =successful 0 =user_skipped 0 =auto_skipped 1 3) erl -s ts install -s ts run all_tests -s init stop # erl -s ts install -s ts run all_tests -s init stop Erlang/OTP 19 [erts-8.2] [source] {"init terminating in do_boot",{undef,[{ts,install,[],[]},{init,start_em,1,[]},{init,do_boot,3,[]}]}} init terminating in do_boot () Crash dump is being written to: erl_crash.dump...done =erl_crash_dump:0.3 Tue Jan 17 11:41:15 2017 Slogan: init terminating in do_boot () System version: Erlang/OTP 19 [erts-8.2] [source] Compiled: Tue Jan 17 11:04:33 2017 Taints: erl_tracer Atoms: 6506 Calling Thread: scheduler:1 =scheduler:1 Scheduler Sleep Info Aux Work: Current Port: Run Queue Max Length: 0 Run Queue High Length: 0 Run Queue Normal Length: 1 Run Queue Low Length: 0 Run Queue Port Length: 0 Run Queue Flags: NONEMPTY_NORMAL | OUT_OF_WORK | HALFTIME_OUT_OF_WORK | NONEMPTY | UNKNOWN(134217728) Current Process: <0.0.0> Current Process State: Running Current Process Internal State: ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING | TRAP_EXIT | ON_HEAP_MSGQ Current Process Program counter: 0x202993d4 (init:boot_loop/2 + 28) Current Process CP: 0x00000000 (invalid) Current Process Limited Stack Trace: 0x20429544:SReturn addr 0x20088FDC () ... 15000 lines Le 16/01/2017 ? 22:25, Kostis Sagonas a ?crit : On 01/16/2017 05:12 PM, REIX, Tony wrote: After building with Hipe, I've tried without Hipe. And I got the same result: an infinite loop, in read() from signal_dispatcher_thread_func(). In: ERLC ../ebin/hipe_rtl.beam with Hipe (--enable-hipe) In: ERLC ../ebin/hipe_consttab.beam without Hipe (--disable-hipe) . No idea why "hipe" appears here also. So, for now, I have a common issue, with or without hipe. I can provide some info regarding the behavior you are experiencing: - Certain modules of the 'hipe' application, though not all. are compiled independently of whether HiPE is enabled or not because they are used by 'dialyzer'. - It just so happens that 'hipe' is the first application that is compiled in the bootstrapping process, so the infinite loop you are getting is not related to HiPE but shows some deeper problem in the system you have built. Bottomline: what you are experiencing is not HiPE-related. Your system is broken. Kostis -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Tue Jan 17 14:40:38 2017 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 17 Jan 2017 14:40:38 +0100 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <5b691c56-5f57-c1cb-1115-a0d1563b20a4@atos.net> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> <3c68e5c3-b6ec-da17-2e2a-aacacff1b037@atos.net> <9ee1a3ca-b0fe-e65d-6b5f-ae415c23eb52@cs.ntua.gr> <623cfbe6-eb50-f3a3-c983-7840263f8131@atos.net> <5b691c56-5f57-c1cb-1115-a0d1563b20a4@atos.net> Message-ID: There are instructions here: https://github.com/erlang/otp/blob/maint/HOWTO/TESTING.md on how to run the tests. Looking at the log it seems like you are executing erl -s ts install ..... in the wrong directory. You should be in release/tests/test_server for erl to find all the correct OTP test libraries. Lukas On Tue, Jan 17, 2017 at 2:18 PM, REIX, Tony wrote: > Resending without the attachment. > > Tony > > Le 17/01/2017 ? 11:52, Tony REIX a ?crit : > > Hi, > > With : > --disable-hipe --disable-threads --disable-smp-support > --without-javac > > I've been able to build Erlang and to launch the tests by: *ct_run > -suite ../compiler_test/andor_SUITE -case t_orelse* > > However, it fails at beginning: "Failed to start CTH", and it talks about > a file "see the CT Log for details"" that I have no idea where it is. > > I've only found at: otp_src_19.2/release/tests/test_server : > ./ct_run.ct@REDACTED/tests.compiler_test. > andor_SUITE.t_orelse.logs/run.2017-01-17_11.14.50/suite.log > but it does not help a lot > > Any idea which document/file I should have a look at in order to > understand the root cause of the failing testing ? > Any idea about the root cause of the failure ? > > > I've also tried with : *erl -s ts install -s ts run all_tests -s init > stop* . > > But it crashes: init terminating in do_boot > > Crash dump > > Looking at: erl_crash.dump , I see: Current Process CP: 0x00000000 ( > *invalid*) > > *erl_crash.dump file is attached to this email.* REMOVED > > > I guess that I have to learn a little bit ( ;) ) more about how Erlang > works. > > See below for details. > > > Thanks > > Tony > > > *1) Launch tests :* > > + cd /opt/freeware/src/packages/BUILD/otp_src_19.2/release/ > tests//test_server > + /opt/freeware/src/packages/BUILD/otp_src_19.2/*bin/ct_run -suite > ../compiler_test/andor_SUITE -case t_orelse* > > Common Test starting (cwd is /opt/freeware/src/packages/ > BUILD/otp_src_19.2/release/tests/test_server) > > Eshell V8.2 (abort with ^G) > (ct@REDACTED)1> > Common Test: Running make in test directories... > (ct@REDACTED)1> Recompile: andor_SUITE > (ct@REDACTED)1> andor_SUITE.erl:83: Warning: this clause cannot match > because of different types/sizes > (ct@REDACTED)1> andor_SUITE.erl:86: Warning: this clause cannot match > because a previous clause always matches > (ct@REDACTED)1> andor_SUITE.erl:87: Warning: this clause cannot match > because a previous clause always matches > (ct@REDACTED)1> andor_SUITE.erl:92: Warning: this clause cannot match > because of different types/sizes > (ct@REDACTED)1> andor_SUITE.erl:169: Warning: the guard for this clause > evaluates to 'false' > (ct@REDACTED)1> andor_SUITE.erl:170: Warning: the guard for this clause > evaluates to 'false' > (ct@REDACTED)1> andor_SUITE.erl:170: Warning: the result of the expression > is ignored (suppress the warning by assigning the expression to the _ > variable) > (ct@REDACTED)1> andor_SUITE.erl:171: Warning: the guard for this clause > evaluates to 'false' > (ct@REDACTED)1> andor_SUITE.erl:172: Warning: the guard for this clause > evaluates to 'false' > (ct@REDACTED)1> andor_SUITE.erl:179: Warning: this expression will fail > with a 'badarg' exception > (ct@REDACTED)1> andor_SUITE.erl:194: Warning: this clause cannot match > because a previous clause at line 194 always matches > (ct@REDACTED)1> andor_SUITE.erl:195: Warning: the guard for this clause > evaluates to 'false' > (ct@REDACTED)1> andor_SUITE.erl:196: Warning: the guard for this clause > evaluates to 'false' > (ct@REDACTED)1> andor_SUITE.erl:197: Warning: the guard for this clause > evaluates to 'false' > (ct@REDACTED)1> andor_SUITE.erl:227: Warning: this clause cannot match > because a previous clause at line 227 always matches > (ct@REDACTED)1> andor_SUITE.erl:228: Warning: this clause cannot match > because a previous clause at line 228 always matches > (ct@REDACTED)1> andor_SUITE.erl:229: Warning: this clause cannot match > because a previous clause at line 229 always matches > (ct@REDACTED)1> andor_SUITE.erl:230: Warning: the guard for this clause > evaluates to 'false' > (ct@REDACTED)1> Recompile: test_lib > (ct@REDACTED)1> > CWD set to: "/opt/freeware/src/packages/BUILD/otp_src_19.2/release/ > tests/test_server/ct_run.ct@REDACTED" > > (ct@REDACTED)1> > TEST INFO: 1 test(s), 1 case(s) in 1 suite(s) > > (ct@REDACTED)1> Testing tests.compiler_test.andor_SUITE.t_orelse: Starting > test, 1 test cases > (ct@REDACTED)1> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - - > Error detected: "*Failed to start CTH*, see the *CT Log for details*" > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - - > > (ct@REDACTED)1> Testing tests.compiler_test.andor_SUITE.t_orelse: *** > FAILED {andor_SUITE,init_per_suite} *** > (ct@REDACTED)1> Testing tests.compiler_test.andor_SUITE.t_orelse: TEST > COMPLETE, 0 ok, 0 failed, 1 skipped of 1 test cases > > > *2)* ./ct_run.ct@REDACTED/tests.compiler_test. > andor_SUITE.t_orelse.logs/run.2017-01-17_11.14.50/*suite.log* > > === Suite started at 2017-01-17 11:14:50 > === Starting test, 1 test cases > =cases 1 > =user root > =host hardy1 > =hosts hardy1 > =emulator_vsn 8.2 > =emulator beam > =otp_release 19 > =started 2017-01-17 11:14:50 > =case andor_SUITE:init_per_suite > =logfile andor_suite.init_per_suite.html > =started 2017-01-17 11:14:50 > =ended 2017-01-17 11:14:50 > =result failed: "Failed to start CTH, see the CT Log for details", > [{andor_SUITE, > > init_per_suite}] > === *** FAILED {andor_SUITE,init_per_suite} *** > === > =case andor_SUITE:t_orelse > =started 2017-01-17 11:14:50 > =result auto_skipped: init_per_suite failed > === *** Skipping test case #1 {andor_SUITE,t_orelse} *** > === > =case andor_SUITE:end_per_suite > =started 2017-01-17 11:14:50 > =result auto_skipped: init_per_suite failed > === *** Skipping {andor_SUITE,end_per_suite} *** > === TEST COMPLETE, 0 ok, 0 failed, 1 skipped of 1 test cases > > =finished 2017-01-17 11:14:50 > =failed 0 > =successful 0 > =user_skipped 0 > =auto_skipped 1 > > > *3)** erl -s ts install -s ts run all_tests -s init stop* > > # erl -s ts install -s ts run all_tests -s init stop > Erlang/OTP 19 [erts-8.2] [source] > > {"init terminating in do_boot",{undef,[{ts,install,[ > ],[]},{init,start_em,1,[]},{init,do_boot,3,[]}]}} > init terminating in do_boot () > > Crash dump is being written to: erl_crash.dump...done > > > =erl_crash_dump:0.3 > Tue Jan 17 11:41:15 2017 > Slogan: init terminating in do_boot () > System version: Erlang/OTP 19 [erts-8.2] [source] > Compiled: Tue Jan 17 11:04:33 2017 > Taints: erl_tracer > Atoms: 6506 > Calling Thread: scheduler:1 > =scheduler:1 > Scheduler Sleep Info Aux Work: > Current Port: > Run Queue Max Length: 0 > Run Queue High Length: 0 > Run Queue Normal Length: 1 > Run Queue Low Length: 0 > Run Queue Port Length: 0 > Run Queue Flags: NONEMPTY_NORMAL | OUT_OF_WORK | HALFTIME_OUT_OF_WORK | > NONEMPTY | UNKNOWN(134217728) > Current Process: <0.0.0> > Current Process State: Running > Current Process Internal State: ACT_PRIO_NORMAL | USR_PRIO_NORMAL | > PRQ_PRIO_NORMAL | ACTIVE | RUNNING | TRAP_EXIT | ON_HEAP_MSGQ > Current Process Program counter: 0x202993d4 (init:boot_loop/2 + 28) > *Current Process CP: 0x00000000 (invalid)* > Current Process Limited Stack Trace: > 0x20429544:SReturn addr 0x20088FDC () > ... > > 15000 lines > > > > Le 16/01/2017 ? 22:25, Kostis Sagonas a ?crit : > > On 01/16/2017 05:12 PM, REIX, Tony wrote: > > > After building with Hipe, I've tried without Hipe. > And I got the same result: an infinite loop, in read() from > signal_dispatcher_thread_func(). > In: ERLC ../ebin/hipe_rtl.beam with Hipe (--enable-hipe) > In: ERLC ../ebin/hipe_consttab.beam without Hipe (--disable-hipe) > . No idea why "hipe" appears here also. > So, for now, I have a common issue, with or without hipe. > > > I can provide some info regarding the behavior you are experiencing: > > - Certain modules of the 'hipe' application, though not all. are compiled > independently of whether HiPE is enabled or not because they are used by > 'dialyzer'. > > - It just so happens that 'hipe' is the first application that is > compiled in the bootstrapping process, so the infinite loop you are getting > is not related to HiPE but shows some deeper problem in the system you have > built. > > Bottomline: what you are experiencing is not HiPE-related. Your system is > broken. > > Kostis > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchamagne@REDACTED Tue Jan 17 15:47:06 2017 From: bchamagne@REDACTED (Bastien Chamagne) Date: Tue, 17 Jan 2017 15:47:06 +0100 Subject: [erlang-questions] Typespecs of functions that may fail In-Reply-To: <9c866c4f-2c64-b1db-fdf4-3c137df672eb@idmog.com> References: <9c866c4f-2c64-b1db-fdf4-3c137df672eb@idmog.com> Message-ID: Hello, I'm trying to use dialyzer and I don't know how to type functions that are written in "happy path" (may crash). Here's a sample function that may or may not crash depending if Field correspond to a relation field on given Model: -spec is_relation(field(), model()) -> {true, relation_type()} | false. %% @doc Sets a relation on the given model. -spec set_relation(field(), model(), model()) -> model() | no_return(). set_relation(Field, RelatedModel, Model) -> % On next line, I'm OK that this function crashes if I try to set a relation on a non-relation field {true, RelationType} = is_relation(Field, Model), set_relation(Field, RelatedModel, Model, RelationType). Here's what dialyzer says: model.erl:180: Function set_relation/3 has no local return model.erl:181: The pattern {'true', RelationType} can never match the type 'false' Is there anything that I can do to inform dialyzer that this is a OK behaviour for me? Cheers! /ps: I'm using erlang 17 if that matters./ /ps: Sorry if it's a duplicate, I can't find the previous message on the archive./ -------------- next part -------------- An HTML attachment was scrubbed... URL: From aronisstav@REDACTED Tue Jan 17 16:08:28 2017 From: aronisstav@REDACTED (Stavros Aronis) Date: Tue, 17 Jan 2017 16:08:28 +0100 Subject: [erlang-questions] Typespecs of functions that may fail In-Reply-To: References: <9c866c4f-2c64-b1db-fdf4-3c137df672eb@idmog.com> Message-ID: Hi Bastien, can it be the case that "is_relation/2" is only used "locally" (i.e. within the module itself) and all such calls to "is_relation/2" will return 'false'? This looks like a case where dialyzer has inferred such a thing and thus complains about the pattern that can't match. Regards, Stavros On Tue, Jan 17, 2017 at 3:47 PM, Bastien Chamagne wrote: > Hello, > > I'm trying to use dialyzer and I don't know how to type functions that are > written in "happy path" (may crash). > Here's a sample function that may or may not crash depending if Field > correspond to a relation field on given Model: > > -spec is_relation(field(), model()) -> {true, relation_type()} | false. > > %% @doc Sets a relation on the given model. > -spec set_relation(field(), model(), model()) -> model() | no_return(). > set_relation(Field, RelatedModel, Model) -> > % On next line, I'm OK that this function crashes if I try to set > a relation on a non-relation field > {true, RelationType} = is_relation(Field, Model), > set_relation(Field, RelatedModel, Model, RelationType). > > Here's what dialyzer says: > > model.erl:180: Function set_relation/3 has no local return > model.erl:181: The pattern {'true', RelationType} can never match the > type 'false' > > Is there anything that I can do to inform dialyzer that this is a OK > behaviour for me? > > Cheers! > > *ps: I'm using erlang 17 if that matters.* > *ps: Sorry if it's a duplicate, I can't find the previous message on the > archive.* > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchamagne@REDACTED Tue Jan 17 16:22:47 2017 From: bchamagne@REDACTED (Bastien Chamagne) Date: Tue, 17 Jan 2017 16:22:47 +0100 Subject: [erlang-questions] Typespecs of functions that may fail In-Reply-To: References: <9c866c4f-2c64-b1db-fdf4-3c137df672eb@idmog.com> Message-ID: <182de63e-a1c7-7656-c9bd-36be96068385@idmog.com> Thank you for the quick reply Stavros! You mean that dialyzer inference is stronger than the defined typespec? On 01/17/2017 04:08 PM, Stavros Aronis wrote: > Hi Bastien, > > can it be the case that "is_relation/2" is only used "locally" (i.e. > within the module itself) and all such calls to "is_relation/2" will > return 'false'? > > This looks like a case where dialyzer has inferred such a thing and > thus complains about the pattern that can't match. > > Regards, > > Stavros > > On Tue, Jan 17, 2017 at 3:47 PM, Bastien Chamagne > wrote: > > Hello, > > I'm trying to use dialyzer and I don't know how to type functions > that are written in "happy path" (may crash). > Here's a sample function that may or may not crash depending if > Field correspond to a relation field on given Model: > > -spec is_relation(field(), model()) -> {true, relation_type()} > | false. > > %% @doc Sets a relation on the given model. > -spec set_relation(field(), model(), model()) -> model() > | no_return(). > set_relation(Field, RelatedModel, Model) -> > % On next line, I'm OK that this function crashes if I try > to set a relation on a non-relation field > {true, RelationType} = is_relation(Field, Model), > set_relation(Field, RelatedModel, Model, RelationType). > > Here's what dialyzer says: > > model.erl:180: Function set_relation/3 has no local return > model.erl:181: The pattern {'true', RelationType} can never > match the type 'false' > > Is there anything that I can do to inform dialyzer that this is a > OK behaviour for me? > > Cheers! > > /ps: I'm using erlang 17 if that matters./ > /ps: Sorry if it's a duplicate, I can't find the previous message > on the archive./ > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aronisstav@REDACTED Tue Jan 17 16:31:22 2017 From: aronisstav@REDACTED (Stavros Aronis) Date: Tue, 17 Jan 2017 15:31:22 +0000 Subject: [erlang-questions] Typespecs of functions that may fail In-Reply-To: <182de63e-a1c7-7656-c9bd-36be96068385@idmog.com> References: <9c866c4f-2c64-b1db-fdf4-3c137df672eb@idmog.com> <182de63e-a1c7-7656-c9bd-36be96068385@idmog.com> Message-ID: Yes. Dialyzer collects all local calls to a non-exported function and analyses its body with the additional constraint that the arguments given to the function are only the ones in those calls. In such cases it may be able to infer types that are stronger than the relevant specs (you can get a warning about the spec itself by using the "-Wunderspecs" option). You can eliminate the warning by e.g. exporting 'is_relation/2' or by using it locally in situations where it may also return `{true, ...}`. Cheers! Stavros On Tue, Jan 17, 2017 at 4:22 PM Bastien Chamagne wrote: > Thank you for the quick reply Stavros! > > You mean that dialyzer inference is stronger than the defined typespec? > > > On 01/17/2017 04:08 PM, Stavros Aronis wrote: > > Hi Bastien, > > can it be the case that "is_relation/2" is only used "locally" (i.e. > within the module itself) and all such calls to "is_relation/2" will return > 'false'? > > This looks like a case where dialyzer has inferred such a thing and thus > complains about the pattern that can't match. > > Regards, > > Stavros > > On Tue, Jan 17, 2017 at 3:47 PM, Bastien Chamagne > wrote: > > Hello, > > I'm trying to use dialyzer and I don't know how to type functions that are > written in "happy path" (may crash). > Here's a sample function that may or may not crash depending if Field > correspond to a relation field on given Model: > > -spec is_relation(field(), model()) -> {true, relation_type()} | false. > > %% @doc Sets a relation on the given model. > -spec set_relation(field(), model(), model()) -> model() | no_return(). > set_relation(Field, RelatedModel, Model) -> > % On next line, I'm OK that this function crashes if I try to set > a relation on a non-relation field > {true, RelationType} = is_relation(Field, Model), > set_relation(Field, RelatedModel, Model, RelationType). > > Here's what dialyzer says: > > model.erl:180: Function set_relation/3 has no local return > model.erl:181: The pattern {'true', RelationType} can never match the > type 'false' > > Is there anything that I can do to inform dialyzer that this is a OK > behaviour for me? > > Cheers! > > *ps: I'm using erlang 17 if that matters.* > *ps: Sorry if it's a duplicate, I can't find the previous message on the > archive.* > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Tue Jan 17 16:52:12 2017 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 17 Jan 2017 16:52:12 +0100 Subject: [erlang-questions] Typespecs of functions that may fail In-Reply-To: References: <9c866c4f-2c64-b1db-fdf4-3c137df672eb@idmog.com> Message-ID: <3705a027-bceb-e710-8189-69406cbeff3e@cs.ntua.gr> On 01/17/2017 03:47 PM, Bastien Chamagne wrote: > Hello, > > I'm trying to use dialyzer and I don't know how to type functions that > are written in "happy path" (may crash). Dialyzer's core (known as "success typings") are designed for _exactly_ this purpose. So that you only specify what's happening in the "happy paths", i.e., in the paths that a function returns ("succeeds"). So you cannot (and should not) specify anything regarding the "sad paths." > Here's a sample function that may or may not crash depending if Field > correspond to a relation field on given Model: > > -spec is_relation(field(), model()) -> {true, relation_type()} | false. > > %% @doc Sets a relation on the given model. > -spec set_relation(field(), model(), model()) -> model() | no_return(). > set_relation(Field, RelatedModel, Model) -> > % On next line, I'm OK that this function crashes if I try to > set a relation on a non-relation field > {true, RelationType} = is_relation(Field, Model), > set_relation(Field, RelatedModel, Model, RelationType). > > Here's what dialyzer says: > > model.erl:180: Function set_relation/3 has no local return > model.erl:181: The pattern {'true', RelationType} can never match > the type 'false' > > Is there anything that I can do to inform dialyzer that this is a OK > behaviour for me? This behaviour should NOT be OK for you. Dialyzer is telling you that it has somehow discovered that it your is_relation/2 function call always returns 'false' here. Unless your version of Dialyzer has some bug, which is unlikely, you cannot ignore that this warning. In any case, the problem is not in the set_relation/2 function but in the is_relation/2 function for which you have not showed us its code; actually, you need to show us the whole module in order to tell you with certaintly what's happening. Kostis PS. Types of the form type() | no_return() are equally verbose as writing 42 + 0 for the integer 42 ... From tony.reix@REDACTED Tue Jan 17 17:38:46 2017 From: tony.reix@REDACTED (REIX, Tony) Date: Tue, 17 Jan 2017 16:38:46 +0000 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> Message-ID: <93190fd2-4bf6-91ef-3fce-c0de03b12f3d@atos.net> Hi Andras, Thanks for your information ! :) Thanks for the list of minimal features (SMP, 64bit, SSL). Once I'm done with running correctly the tests with the minimal case I have now, I'll move forward and I will enable them. For now, I'm using : --disable-hipe --disable-threads --disable-smp-support --without-javac . And that already does not work since tests fail at beginning. About the assembler issue with 1: labels, my colleague already provided me with a solution, but more complex than yours. I've asked him to explain me what the impact is to use his solution or yours. HIPE: OK. I'll go back to it later. SMP: As I said, I have changed some parts of the assembler, including the erts/include/internal/ppc32/atomic.h file. However, I have no idea if my changes are correct... since I'm now not using SMP. I'll see at the 2 pages you have provided, [1] & [2] . About "scheduler binding", I'll have to check that the way I build Erlang (by means of a FedoraCore25 modified .spec file) does generate all the appropriate flags. For now, I am in 32bit. For now, I see that some configure and Makefile (erts/emulator/powerpc-ibm-aix6.1/Makefile) files contain : DEXPORT = -Wl,-bexpall,-brtl . However, I do not see them in the traces of the build. However, "make" command hides the details. -b64 and -maix64 should come when I am in 64bit. I'll check. Though I've been able to build Erlang till the end, the traces do not show any "./otp_build configure". However, as I said, "make" only shows a summary of what it does. And the --trace option is much too verbose. Applications: it looks that 38 applications where built: asn1 compiler cosEvent cosEventDomain cosFileTransfer cosNotification cosProperty cosTime cosTransactions crypto diameter edoc eldap erl_docgen erl_interface eunit gs hipe ic inets kernel mnesia odbc orber os_mon otp_mibs parsetools percept public_key runtime_tools sasl snmp ssh ssl stdlib syntax_tools tools xmerl Any important application missing ? OTHER: -D_LINUX_SOURCE_COMPAT : YEs !! For sure. We use it sometimes. And I forgot to try with it. I'm rebuilding now with it. Bad news: I see no improvement for my test crash. - clock/time handling issues ? OK. I'll think of this when looking at issues. - vsyslog : I've temporaly worked-around this. I'll go back to it later. - OpenSSL & src.rpm : Yes. We deliver always (or nearly always) the .src.rpm file on BullFreeware. BASIC TESTING: # erl Erlang/OTP 19 [erts-8.2] [source] Eshell V8.2 (abort with ^G) 1> :) That begins to work ! :) - SMP & core/thread number: since, for now, I'm building without SMP, I'll check this later. - Binding schedulers: I'll go back to this later. - erl > crypto:start() : OK : will check. Done: 2> crypto:start(). ok - 3> erlang:system_info(cpu_topology). undefined (No SMP yet) - test suites for the emulator. Hummm I've been able to find some instructions for running tests. No idea for now if these tests include the emulator. I'll look at this once by re-building of Erlang is done.* - # ./otp_build tests : Looks like there is a mismatch in the naming: Makefile:248: /opt/freeware/src/packages/BUILD/otp_src_19.2/make/powerpc-ibm-aix6.1.9.0/otp_ded.mk: No such file or directory # find . -name otp_ded.mk ./make/powerpc-ibm-aix6.1/otp_ded.mk # ln -s /opt/freeware/src/packages/BUILD/otp_src_19.2/make/powerpc-ibm-aix6.1 /opt/freeware/src/packages/BUILD/otp_src_19.2/make/powerpc-ibm-aix6.1.9.0 With this symlink, the ./otp_build tests command works fine, doing a lot of things, and ending without error message. However, then: # cd release/tests/test_server # erl ... 1> ts:install(). Running /opt/freeware/src/packages/BUILD/otp_src_19.2/release/tests/test_server/configure Env: [] configure: error: cannot find sources (conf_vars.in) in /opt/freeware/src/packages/BUILD/otp_src_19.2/release/tests/test_server or .. {error,enoent} It seems that a file is missing. With : cp ./lib/common_test/test_server/conf_vars.in /opt/freeware/src/packages/BUILD/otp_src_19.2/release/tests/test_server that works better. However, the ERL ts:install(). command fails with: Running /opt/freeware/src/packages/BUILD/otp_src_19.2/release/tests/test_server/configure Env: [] ... configure: error: don't know how to compile and link dynamic drivers Salt ? We have very good salt in France ! I'll use ! Thanks a lot for your help !!! And expect that it will take some time before I manage all the cases you have detailed for me. First, I need to have Erlang to work with the minimum, with good tests results. Then, I'll add the removed features, one by one. Regards, Tony Le 15/01/2017 ? 19:43, Boroska Andr?s a ?crit : Hi Tony, On Friday I was too quick to reply and among others I missed the part that you are from Bull. :) I used your R1502 version as the starting point. Still, let me add a few more hints. Mind you my memory can fail (I was working with AIX ~ 2 years ago). Hopefully the experts will correct me if I got something wrong. View it as my 2 cents. It is not comprehensive as you will realise it soon. > ?what are the main features that must be made available ?? AIX users tend to be large banks and similar organisations, and I imagine many of them interested in running RabbitMQ. So, based on my limited knowledge on the matter if you can start RabbitMQ that might be good enough for many. To list the minimal features I would say that SMP, 64bit, SSL are very likely. Others readers might add theirs too. HIPE: Hipe is mainly a performance boost for some applications, but many Erlang use cases do not require it and as Mikael said you can ignore it for now. Linux is always there for people preferring performance over AIX features. :) SMP: On the other hand I would argue that enabling smp support is probably a must for most use cases and it is not too difficult to achieve. One of the issues is that the AIX provided Assembler does not support the local labels syntax of the GNU Assembler. So, one either experiments with the GNU Assembler or change the labels and jumps in erts/include/internal/ppc32/atomic.h. See [1] and [2]. One can replace the label ?1:\t? with ?L1%=:\t? and replace the corresponding jump ?bne- 1b\n\t? with ?bne- L1%=\n\t?. Of course use a different label each time (E.g. L1, L2, L3, ...) This might or might not have performance implications, but definitely better than not having SMP support. Since Power8 offers up to 96 simultaneous threads, scheduler binding might be of interest of some potential users. To achieve that replace [3] and [4] with: #include ?sys/thread.h? bindprocessor(BINDTHREAD, thread_self(), (cpu_t) cpu) or something similar. 64 bit and compiler/linker flags: OBJECT_MODE=64 env variable needs to be exported obviously. GCC needs the following flags to link the executable: -Wl,-bexpall,-brtl,-b64 -maix64 For example you might have success bootstrapping by: CFLAGS=?-maix64 -g -O2 -D_LINUX_SOURCE_COMPAT -L/opt/freeware/lib64 -L/opt/freeware/lib? \ LDFLAGS=?-Wl,-bexpall,-brtl,-b64,-blibpath:/opt/freeware/lib64:opt/freeware/lib -maix64? \ AR=?ar -X64? \ ./otp_build autoconf --with-ssl=/opt/freeware --build= --disable-hipe and then the same flags for running ./otp_build configure Optionally you can also put the flags into the respective conf files. :) Maybe some apps will not compile, like os_mon or odbc. Tip to proceed quickly: you can skip any app by placing an empty file called SKIP in their directory. (lib/*/SKIP) OTHER: Note that the C standard allows the malloc(0) system call to return either zero or a pointer that cannot be dereferenced. Unixes typically return a non-zero pointer, but AIX returns 0 by default. The C interface of Erlang (erl_interface) calls malloc(0) for empty strings and checks the return value against 0 to catch allocation errors. To prevent core dumps in such cases it helps to define -D_LINUX_SOURCE_COMPAT which enforces the non-zero pointer returns. Defining it will probably not hurt and users of C drivers will appreciate. As I mentioned R18 reworked clock/time handling. Expect some incompatibilities in that area. The missing vsyslog on AIX can potentially be worked around by using printf. Or perhaps you know a better solution. As openssl versions seems to go obsolete rather quickly, expect that many users will be interested not only in the binary but also in your source rpm package when you upload to BullFreeware, so they can potentially compile and link with a different openssl version. BASIC TESTING: To check if core/thread number detection works start the Erlang shell and note the smp information in the first line. It should match the number of CPU threads you have. For example [smp:8:8]. See [5] Also note the presence of [64-bit] on the same line. Bonus (may not be critical for many) if you can bind schedulers, for example: erl +sct L0-7c0-7 +stbt db +S8:8 then erlang:system_info(scheduler_bindings). returns {0,1,2,....} and erlang:system_info(cpu_topology). returns [{processor,[{core,{logical,0}},{core,{logical,1}},{core,{logical,2}}, ... }]}] While in the shell issue crypto:start(). that should return ok if linking with ssl worked. There are also test suites for the emulator. You can run ./otp_build setup with the above CFLAGS, AR, LDFLAGS, add your bin dir to the PATH and run ./otp_build tests. Change dir to release/tests/test_server, run erl, then ts:install(), ts:run(emulator, [batch]). Take the above information with a grain of salt as I have no access to AIX and not an expert on building Erlang. The OTP team or others might correct my mistakes. Thank you for your work! Best regards, Andras [1] https://gcc.gnu.org/ml/gcc/2005-12/msg00671.html [2] http://stackoverflow.com/questions/3898435/labels-in-gcc-inline-assembly [3] https://github.com/erlang/otp/blob/maint-19/erts/lib_src/common/erl_misc_utils.c#L591 [4] https://github.com/erlang/otp/blob/maint-19/erts/configure.in#L2294 [5] http://erlang.org/doc/man/erl.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchamagne@REDACTED Tue Jan 17 18:15:43 2017 From: bchamagne@REDACTED (Bastien Chamagne) Date: Tue, 17 Jan 2017 18:15:43 +0100 Subject: [erlang-questions] Typespecs of functions that may fail In-Reply-To: <3705a027-bceb-e710-8189-69406cbeff3e@cs.ntua.gr> References: <9c866c4f-2c64-b1db-fdf4-3c137df672eb@idmog.com> <3705a027-bceb-e710-8189-69406cbeff3e@cs.ntua.gr> Message-ID: Makes perfect sense! Thank you Kostis for this explanation. I'll dig deeper. On 01/17/2017 04:52 PM, Kostis Sagonas wrote: > On 01/17/2017 03:47 PM, Bastien Chamagne wrote: >> Hello, >> >> I'm trying to use dialyzer and I don't know how to type functions that >> are written in "happy path" (may crash). > > Dialyzer's core (known as "success typings") are designed for > _exactly_ this purpose. So that you only specify what's happening in > the "happy paths", i.e., in the paths that a function returns > ("succeeds"). So you cannot (and should not) specify anything > regarding the "sad paths." > > >> Here's a sample function that may or may not crash depending if Field >> correspond to a relation field on given Model: >> >> -spec is_relation(field(), model()) -> {true, relation_type()} | >> false. >> >> %% @doc Sets a relation on the given model. >> -spec set_relation(field(), model(), model()) -> model() | >> no_return(). >> set_relation(Field, RelatedModel, Model) -> >> % On next line, I'm OK that this function crashes if I try to >> set a relation on a non-relation field >> {true, RelationType} = is_relation(Field, Model), >> set_relation(Field, RelatedModel, Model, RelationType). >> >> Here's what dialyzer says: >> >> model.erl:180: Function set_relation/3 has no local return >> model.erl:181: The pattern {'true', RelationType} can never match >> the type 'false' >> >> Is there anything that I can do to inform dialyzer that this is a OK >> behaviour for me? > > This behaviour should NOT be OK for you. Dialyzer is telling you that > it has somehow discovered that it your is_relation/2 function call > always returns 'false' here. Unless your version of Dialyzer has some > bug, which is unlikely, you cannot ignore that this warning. > > In any case, the problem is not in the set_relation/2 function but in > the is_relation/2 function for which you have not showed us its code; > actually, you need to show us the whole module in order to tell you > with certaintly what's happening. > > > Kostis > > PS. Types of the form type() | no_return() are equally verbose as > writing 42 + 0 for the integer 42 ... > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From nmarino@REDACTED Tue Jan 17 23:18:42 2017 From: nmarino@REDACTED (Nick Marino) Date: Tue, 17 Jan 2017 17:18:42 -0500 Subject: [erlang-questions] Strange Dialyzer behavior when matching record fields with opaque types Message-ID: Hi all, I recently encountered a rather strange, unexpected Dialyzer error. I'm unsure if this might be considered a Dialyzer bug, or if I'm just missing some key piece of knowledge that would help me understand what's going on here. I took the code that generated the warning and pruned it down down to this: -module(opaque_weirdness). -export([public_func/0]). -record(a, { d = dict:new() :: dict:dict() }). -record(b, { q = queue:new() :: queue:queue() }). public_func() -> add_element(#b{}, my_key, my_value). add_element(#a{d = Dict}, Key, Value) -> dict:store(Key, Value, Dict); add_element(#b{q = Queue}, Key, Value) -> queue:in({Key, Value}, Queue). Which yields the following warning when I run it through Dialyzer: opaque_weirdness.erl:16: The attempt to match a term of type #b{q::queue:queue(_)} against the pattern {'a', Dict} breaks the opaqueness of queue:queue(_) It seems like this warning is somehow being triggered by the presence of the unused function clause in add_element. If I modify the code and add a line to public_func so that we use both clauses, then Dialyzer passes with no warnings. If I modify the above code to use tagged tuples instead of records (i.e. {dict, Dict} instead of #a{d = Dict}) then it gives a completely different warning, which makes much more sense to me: opaque_weirdness.erl:10: The pattern <{'queue', Queue}, Key, Value> can never match the type <{'dict',dict:dict(_,_)},'my_key','my_value'> So the use of records seems to be necessary to trigger the opaqueness warning, though I can't fathom why this should matter. In the original example code, can anyone explain why Dialyzer is giving a warning about opaqueness instead of producing a "pattern X can never match the type Y" warning? The opaqueness warning seems really bizarre to me; obviously the term will never match the pattern, so how is opaqueness being broken? Thanks, Nick Marino -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Jan 17 23:57:56 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 18 Jan 2017 11:57:56 +1300 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <8dc034b0-1aa4-4b1d-a5f2-a5256bf16f48@Spark> References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> <12f0db3b-989e-a8fa-0017-f0213b27e155@cs.otago.ac.nz> <8dc034b0-1aa4-4b1d-a5f2-a5256bf16f48@Spark> Message-ID: <6bdff689-7c6f-d225-2e99-e74f437e5819@cs.otago.ac.nz> On 17/01/17 8:17 PM, Micha? Muska?a wrote: > > I just checked and all: Python 3, Ruby, Java, C# (so probably F# as > well) and JavaScript properly uppercase the character "?" using the > built-in functions, without any additional libraries. > I didn't use the word "additional". The word *additional* is your word, and is entirely foreign to what I was talking about. Those *ARE* libraries that they are using. I have the source code for the Java 1.8 libraries. java-1.8-src/java/lang/Character.java public static char toUpperCase(char ch) starts at line 6368. This is a library method in a library class. public static int toUpperCase(int ch) starts at line 6397. This is a library method in a library class. It calls a method in CharacterData. java-1.8-src/java/lang/CharacterData.java is an abstract class which dispatches to one of 7 subclasses. java-1.8-src/java/lang/CharacterData00.java handles the Basic Multilingual Plane. int toUpperCase(int ch) starts at line 243. This is a library method for a library method in a library class for a library class. It does some bit fiddling and has a fairly large switch to handle exceptions. If new characters are added to Unicode that need to be exceptions, the code will need rewriting and recompiling. java-1.8/src/java/lang/String.java public String toUppercase(Locale locale) starts at line 2721. The code is rather entertaining, if you are entertained by H. P. Lovecraft. For example, *every* time you call this method, it fetches the name of the locale and checks to see if it is "tr" or "az" or "lt". If another language with case conversion quirks is added, the code will need rewriting and recompiling. The locale-dependent code keeps on calling out to a method in ConditionalSpecialCasing. Well, actually two methods get called for each character in general: one to get a single character, one to get an array of characters. At any rate, this is a library method in a library class. No, I am not going to dig into ConditionalSpecialCasing. These aren't *additional* libraries any more than the unicode module in Erlang is an *additional* library in Erlang. The Java library classes are plain Java code compiled by the plain Java compiler, which knows nothing about how they work. The Erlang modules for handling text are plain Erlang code compiled by the plain Erlang compiler, which knows nothing about how they work. Do note, though, that there are an abstract class with 7 subclasses and another concrete class that are there to support Character and String, which users normally should not use. Note also that while CharacterData{00,01,02,0E,Latin1} were generated automatically, so are presumably up to date, ConditionalSpecialCasing, which handles Greek, Turkish, Azeri, and Lithuanian, was *not*, and its date is 2013. So it may well not be up to date. The same is true of Character (2013). I have no confidence that it's up to date. And while ConditionalSpecialCasing has data for Greek, I note that String doesn't consider Greek to be one of the languages that needs it... It really is *useful* for Unicode processing to be in library files that can be automatically regenerated from the Unicode data base. If you want to really understand case conversion in Unicode, spend a couple of hours figuring out exactly how String.toUpperCase() is done in Java. It's so hairy that they spend an extra pass over the string trying NOT to do it. From kostis@REDACTED Wed Jan 18 00:07:07 2017 From: kostis@REDACTED (Kostis Sagonas) Date: Wed, 18 Jan 2017 00:07:07 +0100 Subject: [erlang-questions] Strange Dialyzer behavior when matching record fields with opaque types In-Reply-To: References: Message-ID: <0f4920ea-11e2-4226-5419-71832809f496@cs.ntua.gr> On 01/17/2017 11:18 PM, Nick Marino wrote: > > I recently encountered a rather strange, unexpected Dialyzer error. I'm > unsure if this might be considered a Dialyzer bug, or if I'm just > missing some key piece of knowledge that would help me understand what's > going on here. > > I took the code that generated the warning and pruned it down down to this: > > -module(opaque_weirdness). > -export([public_func/0]). > > -record(a, { > d = dict:new() :: dict:dict() > }). > > -record(b, { > q = queue:new() :: queue:queue() > }). > > public_func() -> > add_element(#b{}, my_key, my_value). > > add_element(#a{d = Dict}, Key, Value) -> > dict:store(Key, Value, Dict); > add_element(#b{q = Queue}, Key, Value) -> > queue:in({Key, Value}, Queue). > > Which yields the following warning when I run it through Dialyzer: > > opaque_weirdness.erl:16: The attempt to match a term of type > #b{q::queue:queue(_)} against the pattern {'a', Dict} breaks the > opaqueness of queue:queue(_) > > It seems like this warning is somehow being triggered by the presence of > the unused function clause in add_element. If I modify the code and add > a line to public_func so that we use both clauses, then Dialyzer passes > with no warnings. Your analysis is pretty correct. The issue is related to dead code and the warning you get is indeed confusing and weird. This should be fixed in dialyzer. Till then, you can suppress it either by commenting the unused first clause, which is dead code in your module, or by exporting function add_element/3. What roughly happens here is that Dialyzer gets confused by the fact that the first clause will not be used and marks its arguments as having the type none(), i.e. not contributing to the inferred success typing. Then the warning pass sees the none() in the first argument of the clause and thinks it has been produced due to the call to add_element being with a record with a different opaque subterm: these opacity violations are also denoted by/resulting in the none() type. But the pattern matching failure is on the #a{} vs. #b{} level, not on their subterms (fields). Dialyzer _should_ issue a warning for this line, but the correct warning to issue here is something along the lines of: opaque_weirdness.erl:16: The attempt to match a term of type #b{q::queue:queue(_)} against the pattern {'a', Dict} will not succeed or that this clause will not be used since the success typing arguments of this function are (#b{}, _, _) Kostis From michal@REDACTED Wed Jan 18 12:14:09 2017 From: michal@REDACTED (=?utf-8?Q?Micha=C5=82_Muska=C5=82a?=) Date: Wed, 18 Jan 2017 12:14:09 +0100 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <6bdff689-7c6f-d225-2e99-e74f437e5819@cs.otago.ac.nz> References: <58ce3a89-46e8-0e13-c668-298498dfb83e@cs.otago.ac.nz> <12f0db3b-989e-a8fa-0017-f0213b27e155@cs.otago.ac.nz> <8dc034b0-1aa4-4b1d-a5f2-a5256bf16f48@Spark> <6bdff689-7c6f-d225-2e99-e74f437e5819@cs.otago.ac.nz> Message-ID: On 17 Jan 2017, 23:58 +0100, Richard A. O'Keefe , wrote: > > I didn't use the word "additional". > The word *additional* is your word, and is entirely > foreign to what I was talking about. > > Those *ARE* libraries that they are using. > You are completely right, I'm sorry. I haven't considered the standard library, a library. This was obviously wrong of me. When it comes to unicode and locale-specific casing, the unicode standard itself, does not provide them. It provides a language agnostic casing rules with an option for extensions to handle specific language needs. Elixir implements those casing rules directly from the unicode standard (using the provided files). It handles all of the casing conversions form UnicodeData.txt and SpecialCasing.txt. The whole implementation for upcase, downcase and titlecase is about 100 LoC?https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex#L253-L358 Of course it doesn't handle all the corner cases and will not cut it if you need locale-specific rules, but it's compliant with the unicode standard and in practice there are few cases, where you'd need looking beyond it. Micha?. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.reix@REDACTED Wed Jan 18 17:05:17 2017 From: tony.reix@REDACTED (REIX, Tony) Date: Wed, 18 Jan 2017 16:05:17 +0000 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <93190fd2-4bf6-91ef-3fce-c0de03b12f3d@atos.net> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> <93190fd2-4bf6-91ef-3fce-c0de03b12f3d@atos.net> Message-ID: <03ba8156-93bb-b6b0-4648-a4b791b7fe8a@atos.net> Hi, I have this warning: *************************************************** *************************************************** *** WARNING: System might fail to configure or *** might be erroneously configured *** since autoconf version 2.69 is used *** instead of version 2.59! *************************************************** *************************************************** Is it really required to use autoconf v2.59 when building Erlang ? Thanks Tony -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Wed Jan 18 17:10:09 2017 From: kostis@REDACTED (Kostis Sagonas) Date: Wed, 18 Jan 2017 17:10:09 +0100 Subject: [erlang-questions] Port of Erlang (19.2) on AIX In-Reply-To: <03ba8156-93bb-b6b0-4648-a4b791b7fe8a@atos.net> References: <92401602-d485-c1c6-1da8-9ea1460d834f@atos.net> <9a28a95a-d881-4428-df46-63d7f0661855@atos.net> <22651.18370.454189.186497@gargle.gargle.HOWL> <93190fd2-4bf6-91ef-3fce-c0de03b12f3d@atos.net> <03ba8156-93bb-b6b0-4648-a4b791b7fe8a@atos.net> Message-ID: <57ebd99a-67be-cc95-ccb6-9b9451c28506@cs.ntua.gr> On 01/18/2017 05:05 PM, REIX, Tony wrote: > Is it really required to use autoconf v2.59 when building Erlang ? > No. You can safely ignore this warning. Kostis From nmarino@REDACTED Wed Jan 18 17:24:04 2017 From: nmarino@REDACTED (Nick Marino) Date: Wed, 18 Jan 2017 11:24:04 -0500 Subject: [erlang-questions] Strange Dialyzer behavior when matching record fields with opaque types In-Reply-To: <0f4920ea-11e2-4226-5419-71832809f496@cs.ntua.gr> References: <0f4920ea-11e2-4226-5419-71832809f496@cs.ntua.gr> Message-ID: Hi Kostis, Thanks for the quick and helpful response! Glad I'm not crazy after all :). That all makes sense, and in my original use case that uncovered this, I'm fine with just commenting out the unused code for now. I also just uncovered a related(?) issue, in case anyone is interested: in my example code, I found that changing #b{} to #a{} in the call to add_element actually causes Dialyzer to crash. Here's the error message: Proceeding with analysis...{"init terminating in do_boot",{function_clause,[{dialyzer,message_to_string,[{opaque_match,["pattern <{'b', Queue}, Key, Value>","<#a{d::dict:dict(_,_)},'my_key','my_value'>"]}],[{file,"dialyzer.erl"},{line,310}]},{dialyzer,format_warning,2,[{file,"dialyzer.erl"},{line,300}]},{dialyzer_cl,'-print_warnings/1-lc$^0/1-0-',2,[{file,"dialyzer_cl.erl"},{line,818}]},{dialyzer_cl,print_warnings,1,[{file,"dialyzer_cl.erl"},{line,818}]},{dialyzer_cl,return_value,2,[{file,"dialyzer_cl.erl"},{line,715}]},{dialyzer_cl,do_analysis,4,[{file,"dialyzer_cl.erl"},{line,405}]},{dialyzer,'-cl/1-fun-0-',1,[{file,"dialyzer.erl"},{line,153}]},{dialyzer,doit,1,[{file,"dialyzer.erl"},{line,243}]}]}} I'm tempted to try and debug this myself, but I'm not sure I have the necessary skills and background to safely make changes to a tool like Dialyzer. Figured I should leave this one to the experts ;). (Though if anyone with more experience than myself would like to point me in the right direction, I suppose I could take a stab at it - I just don't want to accidentally end up breaking something I don't understand.) Anyway, thanks again! Nick On Tue, Jan 17, 2017 at 6:07 PM, Kostis Sagonas wrote: > On 01/17/2017 11:18 PM, Nick Marino wrote: > >> >> I recently encountered a rather strange, unexpected Dialyzer error. I'm >> unsure if this might be considered a Dialyzer bug, or if I'm just >> missing some key piece of knowledge that would help me understand what's >> going on here. >> >> I took the code that generated the warning and pruned it down down to >> this: >> >> -module(opaque_weirdness). >> -export([public_func/0]). >> >> -record(a, { >> d = dict:new() :: dict:dict() >> }). >> >> -record(b, { >> q = queue:new() :: queue:queue() >> }). >> >> public_func() -> >> add_element(#b{}, my_key, my_value). >> >> add_element(#a{d = Dict}, Key, Value) -> >> dict:store(Key, Value, Dict); >> add_element(#b{q = Queue}, Key, Value) -> >> queue:in({Key, Value}, Queue). >> >> Which yields the following warning when I run it through Dialyzer: >> >> opaque_weirdness.erl:16: The attempt to match a term of type >> #b{q::queue:queue(_)} against the pattern {'a', Dict} breaks the >> opaqueness of queue:queue(_) >> >> It seems like this warning is somehow being triggered by the presence of >> the unused function clause in add_element. If I modify the code and add >> a line to public_func so that we use both clauses, then Dialyzer passes >> with no warnings. >> > > Your analysis is pretty correct. The issue is related to dead code and > the warning you get is indeed confusing and weird. This should be fixed in > dialyzer. Till then, you can suppress it either by commenting the unused > first clause, which is dead code in your module, or by exporting function > add_element/3. > > > What roughly happens here is that Dialyzer gets confused by the fact that > the first clause will not be used and marks its arguments as having the > type none(), i.e. not contributing to the inferred success typing. Then the > warning pass sees the none() in the first argument of the clause and thinks > it has been produced due to the call to add_element being with a record > with a different opaque subterm: these opacity violations are also denoted > by/resulting in the none() type. But the pattern matching failure is on > the #a{} vs. #b{} level, not on their subterms (fields). > > Dialyzer _should_ issue a warning for this line, but the correct warning > to issue here is something along the lines of: > > opaque_weirdness.erl:16: The attempt to match a term of type > #b{q::queue:queue(_)} against the pattern {'a', Dict} will not succeed > > or that this clause will not be used since the success typing arguments of > this function are (#b{}, _, _) > > Kostis > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vans_163@REDACTED Wed Jan 18 19:40:36 2017 From: vans_163@REDACTED (Vans S) Date: Wed, 18 Jan 2017 18:40:36 +0000 (UTC) Subject: [erlang-questions] gen_statem confusion References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> Message-ID: <1798520640.8723249.1484764836743@mail.yahoo.com> What is the correct way to have an event proc next in the queue before any other events/actions are procced. Meaning this event should come before any send / cast / call that were inserted into the process mailbox WHILE the event was being handled. The options seem to be returning a timeout of 0 or returning next_event. It is confusing which one is correct to use to get the desired behavior. Example: handle_event(timeout, _, _, _) -> long_running_func(), %a proc sends to us send(self(), msg) {next_event, internal, next_state}, %OR {next_state, next_state, no_data, 0}. We want to proc the next_state event/timeout befor handing the msg msg we got from a random process. From mononcqc@REDACTED Wed Jan 18 20:20:25 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 18 Jan 2017 14:20:25 -0500 Subject: [erlang-questions] gen_statem confusion In-Reply-To: <1798520640.8723249.1484764836743@mail.yahoo.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> Message-ID: <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> On 01/18, Vans S wrote: >Example: > >handle_event(timeout, _, _, _) -> > long_running_func(), > %a proc sends to us send(self(), msg) > {next_event, internal, next_state}, > %OR > > {next_state, next_state, no_data, 0}. > >We want to proc the next_state event/timeout befor handing the msg msg >we got >from a random process. The {next_state, StateName, Data, Actions} form allows `Actions' to be one or more of: - `postpone', which will replay the current event on the next state transition and skip it for now. This one has to do with replicating a selective receive - `{next_event, EventType, EventContent}', which will enqueue the generated event as the next one to run no matter what, unless other events were already enqueued with the {next_event, ...} action beforehand. This is a mechanism to allow self-driving state machines with event injection without losing the ability to trace or handle calls from the sys module (i.e. the state machine does not become unresponsive) - 'enter action' events, related to handling state transitions. Those include timeouts (generate a message if no event has been received in a while), state timeouts (generate a message if no state transition has happened in a while), and asking for hibernation. Those have to do with handling the behaviour of the process between events or state transitions. - replying to a process that has previously sent a synchronous call So if you want the next event run internally to be 'next_state' (which I assume means "progress to the next state" in your state machine), you'd need to return: handle_event(timeout, _TimeoutMsg, StateName, Data) -> ... {next_state, StateName, Data, [{next_event, internal, next_state}]}. which will enqueue 'next_state' as an internally-generated message to be handled. Regards, Fred. From vans_163@REDACTED Wed Jan 18 21:27:08 2017 From: vans_163@REDACTED (Vans S) Date: Wed, 18 Jan 2017 20:27:08 +0000 (UTC) Subject: [erlang-questions] gen_statem confusion In-Reply-To: <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> Message-ID: <330814690.8825912.1484771228894@mail.yahoo.com> Thanks for the detailed response I will try it this way. What I want to do is progress through a chain of states without processing any messages in between, and once the final state is reached to start processing messages normally. On Wednesday, January 18, 2017 2:20 PM, Fred Hebert wrote: On 01/18, Vans S wrote: >Example: > >handle_event(timeout, _, _, _) -> > long_running_func(), > %a proc sends to us send(self(), msg) > {next_event, internal, next_state}, > %OR > > {next_state, next_state, no_data, 0}. > >We want to proc the next_state event/timeout befor handing the msg msg >we got >from a random process. The {next_state, StateName, Data, Actions} form allows `Actions' to be one or more of: - `postpone', which will replay the current event on the next state transition and skip it for now. This one has to do with replicating a selective receive - `{next_event, EventType, EventContent}', which will enqueue the generated event as the next one to run no matter what, unless other events were already enqueued with the {next_event, ...} action beforehand. This is a mechanism to allow self-driving state machines with event injection without losing the ability to trace or handle calls from the sys module (i.e. the state machine does not become unresponsive) - 'enter action' events, related to handling state transitions. Those include timeouts (generate a message if no event has been received in a while), state timeouts (generate a message if no state transition has happened in a while), and asking for hibernation. Those have to do with handling the behaviour of the process between events or state transitions. - replying to a process that has previously sent a synchronous call So if you want the next event run internally to be 'next_state' (which I assume means "progress to the next state" in your state machine), you'd need to return: handle_event(timeout, _TimeoutMsg, StateName, Data) -> ... {next_state, StateName, Data, [{next_event, internal, next_state}]}. which will enqueue 'next_state' as an internally-generated message to be handled. Regards, Fred. From kennethlakin@REDACTED Wed Jan 18 21:37:50 2017 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 18 Jan 2017 12:37:50 -0800 Subject: [erlang-questions] gen_statem confusion In-Reply-To: <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> Message-ID: On 01/18/2017 11:20 AM, Fred Hebert wrote: > So if you want the next event run internally to be 'next_state' (which I > assume means "progress to the next state" in your state machine), you'd > need to return: > > handle_event(timeout, _TimeoutMsg, StateName, Data) -> > ... > {next_state, StateName, Data, [{next_event, internal, next_state}]}. > > which will enqueue 'next_state' as an internally-generated message > to be handled. You need to be a little careful with this. next_event is a _stack_, not a queue. If you return {next_state, state, Data, [{next_event, internal, a}, {next_event, internal, b}]} and then in state(internal, a, Data) you return {next_state, state, Data, [{next_event, internal, c}, {next_event, internal, d}]} The next three calls (assuming nothing else is pushed on the next_event stack) will be state(internal, c, Data) state(internal, d, Data) state(internal, b, Data) It's really easy to overlook this. This subtle quirk was the cause of at least one bug in a _widely_ used library. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From mononcqc@REDACTED Wed Jan 18 21:48:21 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 18 Jan 2017 15:48:21 -0500 Subject: [erlang-questions] gen_statem confusion In-Reply-To: References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> Message-ID: <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> On 01/18, Kenneth Lakin wrote: >You need to be a little careful with this. next_event is a _stack_, not >a queue. [...] It's really easy to overlook this. This subtle quirk was >the cause of at least one bug in a _widely_ used library. Specifically I copied the terminology from the docs at http://erlang.org/doc/man/gen_statem.html#type-action : > The stored events are inserted in the queue as the next to process > before any already queued events. The order of these stored events is > preserved, so the first next_event in the containing list becomes the > first to process. I guess that could use some rewording. From vans_163@REDACTED Wed Jan 18 23:03:45 2017 From: vans_163@REDACTED (Vans S) Date: Wed, 18 Jan 2017 22:03:45 +0000 (UTC) Subject: [erlang-questions] gen_statem confusion In-Reply-To: <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> Message-ID: <1858145896.95613.1484777025374@mail.yahoo.com> Would using a timeout or state_timeout of 0 provide the same guarantees as using next_event? Looking at the API it seems state_timeout maybe just shortform for {next_event, state_timeout, ..} On Wednesday, January 18, 2017 3:48 PM, Fred Hebert wrote: On 01/18, Kenneth Lakin wrote: >You need to be a little careful with this. next_event is a _stack_, not >a queue. [...] It's really easy to overlook this. This subtle quirk was >the cause of at least one bug in a _widely_ used library. Specifically I copied the terminology from the docs at http://erlang.org/doc/man/gen_statem.html#type-action : > The stored events are inserted in the queue as the next to process > before any already queued events. The order of these stored events is > preserved, so the first next_event in the containing list becomes the > first to process. I guess that could use some rewording. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From kennethlakin@REDACTED Wed Jan 18 23:18:23 2017 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 18 Jan 2017 14:18:23 -0800 Subject: [erlang-questions] gen_statem confusion In-Reply-To: <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> Message-ID: On 01/18/2017 12:48 PM, Fred Hebert wrote: > Specifically I copied the terminology from the docs at > http://erlang.org/doc/man/gen_statem.html#type-action : > I guess that could use some rewording. Yeah, the gen_statem documentation has some significantly rough patches. (Insert standard "good documentation is hard and often boring work, and anyone who volunteers to write accurate documentation that's even vaguely legible is a minor hero" disclaimer here.) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From kostis@REDACTED Thu Jan 19 01:07:44 2017 From: kostis@REDACTED (Kostis Sagonas) Date: Thu, 19 Jan 2017 01:07:44 +0100 Subject: [erlang-questions] Strange Dialyzer behavior when matching record fields with opaque types In-Reply-To: References: <0f4920ea-11e2-4226-5419-71832809f496@cs.ntua.gr> Message-ID: <47cc6d52-0411-047e-e72b-4c3fd702e4bd@cs.ntua.gr> On 01/18/2017 05:24 PM, Nick Marino wrote: > Hi Kostis, > > Thanks for the quick and helpful response! Glad I'm not crazy after all > :). That all makes sense, and in my original use case that uncovered > this, I'm fine with just commenting out the unused code for now. > > I also just uncovered a related(?) issue, in case anyone is interested: > in my example code, I found that changing #b{} to #a{} in the call to > add_element actually causes Dialyzer to crash. Here's the error message: > > Proceeding with analysis...{"init terminating in > do_boot",{function_clause,[{dialyzer,message_to_string,[{opaque_match,["pattern > <{'b', Queue}, Key, > Value>","<#a{d::dict:dict(_,_)},'my_key','my_value'>"]}],[{file,"dialyzer.erl"},{line,310}]},{dialyzer,format_warning,2,[{file,"dialyzer.erl"},{line,300}]},{dialyzer_cl,'-print_warnings/1-lc$^0/1-0-',2,[{file,"dialyzer_cl.erl"},{line,818}]},{dialyzer_cl,print_warnings,1,[{file,"dialyzer_cl.erl"},{line,818}]},{dialyzer_cl,return_value,2,[{file,"dialyzer_cl.erl"},{line,715}]},{dialyzer_cl,do_analysis,4,[{file,"dialyzer_cl.erl"},{line,405}]},{dialyzer,'-cl/1-fun-0-',1,[{file,"dialyzer.erl"},{line,153}]},{dialyzer,doit,1,[{file,"dialyzer.erl"},{line,243}]}]}} > > I'm tempted to try and debug this myself, but I'm not sure I have the > necessary skills and background to safely make changes to a tool like > Dialyzer. Figured I should leave this one to the experts ;) Thanks again for reporting this issue, which I can reproduce. One can easily avoid the crash by the following patch: ================================================================ diff --git a/lib/dialyzer/src/dialyzer.erl b/lib/dialyzer/src/dialyzer.erl index d25ffd0..d5352fd 100644 --- a/lib/dialyzer/src/dialyzer.erl +++ b/lib/dialyzer/src/dialyzer.erl @@ -438,6 +438,8 @@ message_to_string({opaque_guard, [Arg1, Infix, Arg2, ArgNs]}) -> message_to_string({opaque_guard, [Guard, Args]}) -> io_lib:format("Guard test ~w~s breaks the opacity of its argument\n", [Guard, Args]); +message_to_string({opaque_match, [Pat, Term]}) -> + io_lib:format("The ~s can never match the term ~s\n", [Pat, Term]); message_to_string({opaque_match, [Pat, OpaqueType, OpaqueTerm]}) -> Term = if OpaqueType =:= OpaqueTerm -> "the term"; true -> OpaqueTerm ================================================================ which produces the following warning in your example program: opaque_weirdness.erl:13: The pattern <{'b', Queue}, Key, Value> can never match the term <#a{d::dict:dict(_,_)},'my_key','my_value'> However, this patch is sub-optimal. Similarly to the previous issue you reported the core of the problem is that dialyzer erroneously classifies the problem as opaque-related, which is not. It's a simple case of dead code (a clause that will never match) and the warning should be tagged with 'pattern_match' instead of 'opaque_match' or, better yet, simply say that the 2nd clause is unused (cannot match). I will leave this issue to the responsible developer at Ericsson to handle properly. Kostis PS. The test case is: -module(opaque_weirdness). -export([public_func/0]). -record(a, {d = dict:new() :: dict:dict()}). -record(b, {q = queue:new() :: queue:queue()}). public_func() -> add_element(#a{}, my_key, my_value). add_element(#a{d = Dict}, Key, Value) -> dict:store(Key, Value, Dict); add_element(#b{q = Queue}, Key, Value) -> queue:in({Key, Value}, Queue). From mononcqc@REDACTED Thu Jan 19 01:27:43 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 18 Jan 2017 19:27:43 -0500 Subject: [erlang-questions] gen_statem confusion In-Reply-To: <1858145896.95613.1484777025374@mail.yahoo.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> Message-ID: <20170119002742.GD26705@ferdmbp.local> On 01/18, Vans S wrote: >Would using a timeout or state_timeout of 0 provide the same guarantees as using next_event? >Looking at the API it seems state_timeout maybe just shortform for {next_event, state_timeout, ..} > No. A regular timeout would not trigger if messages are received or waiting in the mailbox A next_event timeout works as follows according to the docs. > If the value is 0 no timer is actually started, instead the the > time-out event is enqueued to ensure that it gets processed before > any not yet received external event. Already received external events and enqueued next_events will likely run before state timeouts. From vans_163@REDACTED Thu Jan 19 04:01:07 2017 From: vans_163@REDACTED (Vans S) Date: Thu, 19 Jan 2017 03:01:07 +0000 (UTC) Subject: [erlang-questions] gen_statem confusion In-Reply-To: <20170119002742.GD26705@ferdmbp.local> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> <20170119002742.GD26705@ferdmbp.local> Message-ID: <634078528.341606.1484794867558@mail.yahoo.com> This wording is really confusing: > instead the the time-out event is enqueued to ensure > that it gets processed before any not yet received > external event Because when I tried a test case init() -> send(self(), hi), {_,_,_,0}. %0 timeout basically The 0 timeout procced before the 'hi' message. Using 1 as the timeout, 'hi' message procs first. But it says not yet received external event. To me send() seems like an external event. Hence confusing. On Wednesday, January 18, 2017 7:27 PM, Fred Hebert wrote: On 01/18, Vans S wrote: >Would using a timeout or state_timeout of 0 provide the same guarantees as using next_event? >Looking at the API it seems state_timeout maybe just shortform for {next_event, state_timeout, ..} > No. A regular timeout would not trigger if messages are received or waiting in the mailbox A next_event timeout works as follows according to the docs. > If the value is 0 no timer is actually started, instead the the > time-out event is enqueued to ensure that it gets processed before > any not yet received external event. Already received external events and enqueued next_events will likely run before state timeouts. From ewdpb@REDACTED Thu Jan 19 04:19:47 2017 From: ewdpb@REDACTED (=?UTF-8?Q?Wilmar_P=C3=A9rez?=) Date: Wed, 18 Jan 2017 22:19:47 -0500 Subject: [erlang-questions] VoIP Recorder on Erlang? Message-ID: Hello all We are developing an application that requires recording segments of VoIP calls. We expect large volumes of calls and we are expected to be able to identify call streams from a heavy spanned pipe (perhaps using libpcap or something similar). I think handling this functionality with Erlang makes a lot of sense. However, I am not yet the biggest Erlang expert. Is there any similar project that I could start working from as a base or at least as a guideline? Any word of advice or recommendation is welcome. Thank you! Wilmar -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.boaventura@REDACTED Thu Jan 19 04:14:55 2017 From: michel.boaventura@REDACTED (Michel Boaventura) Date: Thu, 19 Jan 2017 01:14:55 -0200 Subject: [erlang-questions] Problem with binary_to_term on different machines Message-ID: Hi all, I've build erlang 19 from scratch on my gentoo machine but can't use dialyzer, because it complains my beam files weren't compiled with debug_info. I've read and debugged a log of dialyzer and erlang code and find out the issue is that for some reason erlang can't parse the abstract part of the files. To check this I save the abstract part on a file and tried to parse it on my gentoo machine and on a Ubuntu server I have access. For some reason I don't know it reads with no problem Ubuntu but gives me an invalid arg on gentoo. I'm very new to erlang and have no clue of how to debug further. The binary is attached and I'm reading it like: {ok, [Binary]} = file:consult("Abstr"). Term = binary_to_term(Binary). ps: I've sent a message similar to this one on this list, but since it was my first one I think it wasn't accepted (and I do apologize for the double post if you received it). But after it I've discovered the problem with binary_to_term, so that message isn't so useful anyway. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Thu Jan 19 07:23:32 2017 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 18 Jan 2017 22:23:32 -0800 Subject: [erlang-questions] A math module better than C89 Message-ID: <58805B64.6080704@gmail.com> There was a recent pull request at https://github.com/erlang/otp/pull/1298 which was unable to go further, due to it not being clear how bigint support should exist in the math module. The absence of bigint support has been a long-term annoyance, and the pull request attempted to move custom code from elsewhere in Erlang/OTP to make a custom bigint pow/2 (and pow/3) exposed in the math module. I like the idea of pursuing the principle of least astonishment, and allowing the existing pow function to handle bigints, similar how you don't have to think about the difference in python (and other programming languages that people enjoy math in). However, after determining how to do the API, it would then require work to take place to support bigints, which ideally would just be calls to libgmp functions. So, anyone with strong opinions should respond with the hope that the next source code addition might not get refused. Best Regards, Michael From ok@REDACTED Thu Jan 19 09:28:14 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 19 Jan 2017 21:28:14 +1300 Subject: [erlang-questions] A math module better than C89 In-Reply-To: <58805B64.6080704@gmail.com> References: <58805B64.6080704@gmail.com> Message-ID: <5d72f814-7dc6-bb56-96d8-fc68035ebbc8@cs.otago.ac.nz> I honestly don't think there's much difficulty figuring out what the math functions *should* do. To start with, we cannot offer a math library better than C89 until we offer a math library as good as C89, and we just don't do that. - there is no frexp and that turns out to matter a lot - there is no ldexp - there is no floor (round is provided elsewhere) - there is no ceil (trunc is provided elsewhere) - there is no fmod - there is no modf There's also a function that was in the Unix libraries but didn't make it into the C standard until 1999: - there is no hypot (hypot and atan2 are *partners*). Since it's now 18 years since C99, shouldn't we be thinking about a math module better than C99? (The real part anyway.) Let's review what's in the math module, working from http://erlang.org/doc/man/math.html In working on my Smalltalk library, I wrote support functions to convert a bignum to a normalised fraction in float, double, or long double form and an integer power of 2, rather like frexp. Putting this in Erlang terms, if we have frexp(Number) -> {Fraction, Scale} then log(Number) -> {Fraction, Scale} = frexp(Number), underlying:log(Fraction) + Scale * underlying:log(2.0). This means not having to worry about floating point overflow when dealing with logs of (positive) bignums. As it happens, ANSI Smalltalk does not include a logarithm operation on integers, so I *didn't* do this. Before going to sleep tonight, I will! acos(X) X must be between -1 and 1 inclusive. bignums must be reported as wrong. acosh(X) log(X + (X+1)sqrt((X-1)/(X+1)) (X-1)/(X+1) will be effectively 1, sqrt((X-1)/(X+1)) will be effectively 1, acosh(X) will be approximately log(2X+1). See note about log above. asin(X) X must be between -1 and 1 inclusive. bignums must be reported as wrong. asinh(X) log(X + sqrt(1 + X**2)) X**2 + 1 will be effectively 1, sqrt(X**2 + 1) will be effectively |X|, asinh(X) will be effectively log(2X). See note about log above. atan(X) Bignums that would be converted to ?inf may as well be allowed to, atan(?inf) = ?pi atan2(X, Y) After dealing with X == 0 or Y == 0 specially, remember the signs. We can work with {FX, SX} = frexp(X), {FY, SY} = frexp(Y), FR = FY/FX, SR = SY-SX R = if SR < lower threshold -> 0 ; SR > upper threshold -> +infinity ; true -> ldexp(FR, SR) end Now compute the result from R and the saved signs. atanh(X) tanh(Y) lies between -1 and 1 inclusive, so any integer, big or small, other than -1, 0, 1 is an error. MISSING: cbrt/1. See sqrt. cos(X) To be honest, I don't think this really makes sense. I know that the Sun math library claimed to get as good an answer as you could hope for using "infinitely precise pi". I have no idea how that works. The old UNIX libm interface used to report doubles that were too big as basically not terribly meaningful. (If an ulp is larger than pi, you really cannot represent an angle in any meaningful way. And that happens about 2.8e16. So frankly, if a bignum would convert to infinity, I say let it, and let the C library tell you that cos(inf) is NaN. cosh(X) (exp(X) + exp(-X))/2 will be near exp(|X|). +infinity for a bignum. See exp. exp(X) The result is a float. For any float X greater than about 709.78 the answer will be +infinity. A negative bignum can give the answer 0.0. A positive bignum should give the answer +infinity. MISSING AND IMPORTANT: expm1/1 log(X) Described above. We can do this! log10(X) is log(X)/log(10) log2(X) is log(X)/log(2) MISSING AND IMPORTANT: log1p/1 pow(X, Y) Smalltalk distinguishes between number raisedToInteger: integer (which typically uses the bit oriented square and multiply approach) and base raisedTo: power which reverts to raisedToInteger: if power is an integer otherwise uses exp(log(base) * power). This needs a careful case analysis. pow(Bignum, Float) will of course only be allowed when Bignum > 0. Since we can compute log(Bignum), this reduces to exp(log(Bignum)*Float). Sometimes that will give 0, sometimes a finite number, and sometimes ?infinity. It's all doable by careful case analysis. sin(X) See comment on cos(X). sinh(X) (exp(X) - exp(-X))/2. If bignum X > 0, -> exp(X)/2 -> +infinity. If bignum X < 0, -> -exp(-X)/2 -> -infinity. sqrt(X) Smalltalk actually defines sqrt on integers as integer square root, which is a pain in the posterior. I have wanted the integer square root function occasionally, true. But I have much more often wanted anInteger sqrt -> anInteger asFloat sqrt, which is what Erlang does, hooray, hooray. If bignum X < 0, this is an error. Let {F, S} = frexp(X). Let {G, T} = if S odd -> {sqrt(F*2.0), (S-1) div 2} ; S even -> (sqrt(F), S div 2) end. Answer ldexp(G, T). This will give you good answers up to 3.2x10**616 and then infinity. tan(X) See comment on cos. tanh(X) (exp(X) - exp(-X))/(exp(X) + exp(-X)) For any X > about 709.78, answer +infinity. For any X < about -709.78, answer -infinity. erf(X) From the graph of erf at http://mathworld.wolfram.com/Erf.html it is pretty obvious that For bignum X > 0, erf(X) -> 1.0 and erfc(X) -> 0.0 For bignum X < 0, erf(X) -> -1.0 and erfc(X) -> 2.0 pi() has no argument, so no bignum issue. I presume pow(X, Y, M) computes (X mod M)**Y mod M? This reduces to multiplication modulo M (is that '*'(X, Y, M) ?) which is not problematic. The core here is two operations that have been in the C library (and in many languages under various guises) but are not in the Erlang library: frexp and ldexp. I have a faint recollection of libgmp offering frexp, if not I could donate my C code. Of course many of these things are in other languages with bignums. (Hands up everyone who realised I was staring at page 308 of CLtL2? Which doesn't actually mention bignums.) By the way, my HoD and HR are growling at me for not publishing enough, and a friend this afternoon said "don't be ashamed of publishing simple things". Could I get a paper out of this if I finished the case analyses and wrapped a few more words around it? (Seriously, I am quite stressed about the situation.) From alex0player@REDACTED Thu Jan 19 09:59:37 2017 From: alex0player@REDACTED (Alex S.) Date: Thu, 19 Jan 2017 11:59:37 +0300 Subject: [erlang-questions] gen_statem confusion In-Reply-To: <634078528.341606.1484794867558@mail.yahoo.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> <20170119002742.GD26705@ferdmbp.local> <634078528.341606.1484794867558@mail.yahoo.com> Message-ID: <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> > 19 ???. 2017 ?., ? 6:01, Vans S ???????(?): > > > > This wording is really confusing: > >> instead the the time-out event is enqueued to ensure >> that it gets processed before any not yet received >> external event > Because when I tried a test case > > init() -> > send(self(), hi), > {_,_,_,0}. %0 timeout basically > > The 0 timeout procced before the 'hi' message. > Using 1 as the timeout, 'hi' message procs first. > > But it says not yet received external event. To me send() seems like > an external event. Hence confusing. Well, it was sent, but not yet received by the gen_statem?s main loop. From sperber@REDACTED Thu Jan 19 09:57:54 2017 From: sperber@REDACTED (Michael Sperber) Date: Thu, 19 Jan 2017 09:57:54 +0100 Subject: [erlang-questions] Call for Participation: BOB 2017 (February 24, Berlin) - early-bird ends Jan 23 Message-ID: BOB has a strong focus on functional programming, and, as always, a strong Erlang contingent! ================================================================ BOB 2017 Conference "What happens if we simply use what's best?" February 24, 2017 Berlin http://bobkonf.de/2017/ Program: http://bobkonf.de/2017/program.html Registration: http://bobkonf.de/2017/registration.html ================================================================ BOB is the conference for developers, architects and decision-makers to explore technologies beyond the mainstream in software development, and to find the best tools available to software developers today. Our goal is for all participants of BOB to return home with new insights that enable them to improve their own software development experiences. The program features 14 talks and 8 tutorials on current topics: http://bobkonf.de/2017/program.html The subject range of talks includes functional programming, advanced front-end development, and sophisticated uses of types. The tutorials feature introductions to Haskell, Swift, PureScript, React, QuickCheck, Agda, CRDTs and Servant. John Hughes will give the keynote talk. Registration is open online: http://bobkonf.de/2017/registration.html NOTE: The early-bird rates expire on January 23, 2017! BOB cooperates with the :clojured conference on the following day. There is a registration discount available for participants of both events. http://www.clojured.de/ From raimo+erlang-questions@REDACTED Thu Jan 19 11:13:48 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 19 Jan 2017 11:13:48 +0100 Subject: [erlang-questions] gen_statem confusion In-Reply-To: <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> <20170119002742.GD26705@ferdmbp.local> <634078528.341606.1484794867558@mail.yahoo.com> <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> Message-ID: <20170119101348.GA36166@erix.ericsson.se> On Thu, Jan 19, 2017 at 11:59:37AM +0300, Alex S. wrote: > > > 19 ???. 2017 ?., ? 6:01, Vans S ???????(?): > > > > > > > > This wording is really confusing: > > > >> instead the the time-out event is enqueued to ensure > >> that it gets processed before any not yet received > >> external event > > Because when I tried a test case > > > > init() -> > > send(self(), hi), > > {_,_,_,0}. %0 timeout basically > > > > The 0 timeout procced before the 'hi' message. > > Using 1 as the timeout, 'hi' message procs first. > > > > But it says not yet received external event. To me send() seems like > > an external event. Hence confusing. > Well, it was sent, but not yet received by the gen_statem?s main loop. Perfectly correct. While gen_statem processes enqueued events it is guaranteed to not enter a receive statement, so no new messages are received until after ther state_timeout 0 message. Other enqueued messages that are being processed such as next_event messages will be processed before a state_timeout 0 message. So if you in a state callback insert more next_event messages these are inserted first in the queue, and a state_timeout 0 message is inserted last in the queue. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From silviu.cpp@REDACTED Thu Jan 19 12:51:35 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Thu, 19 Jan 2017 13:51:35 +0200 Subject: [erlang-questions] comparing of atoms in native code Message-ID: hello let's suppose I have a static ERL_NIF_TERM which is equal with an atom. example: ERL_NIF_TERM make_atom(ErlNifEnv* env, const char* name) { ERL_NIF_TERM ret; if(enif_make_existing_atom(env, name, &ret, ERL_NIF_LATIN1)) return ret; return enif_make_atom(env, name); } static ERL_NIF_TERM atomTrue = make_atom(env, "true"); It is safe to compare this with other ERL_NIF_TERM values using = or != operators ? Or I need to use enif_is_identical ? Silviu -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Thu Jan 19 13:24:51 2017 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 19 Jan 2017 13:24:51 +0100 Subject: [erlang-questions] comparing of atoms in native code In-Reply-To: References: Message-ID: Hello, On Thu, Jan 19, 2017 at 12:51 PM, Caragea Silviu wrote: > > It is safe to compare this with other ERL_NIF_TERM values using = or != > operators ? > > Or I need to use enif_is_identical ? > You have to use enif_is_identical. In the current implementation it works to use == or !=, but that may very will change in the future. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Thu Jan 19 14:42:02 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 19 Jan 2017 14:42:02 +0100 Subject: [erlang-questions] gen_statem confusion In-Reply-To: References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> Message-ID: <20170119134202.GB36166@erix.ericsson.se> On Wed, Jan 18, 2017 at 02:18:23PM -0800, Kenneth Lakin wrote: > On 01/18/2017 12:48 PM, Fred Hebert wrote: > > Specifically I copied the terminology from the docs at > > http://erlang.org/doc/man/gen_statem.html#type-action : > > > > > I guess that could use some rewording. > > Yeah, the gen_statem documentation has some significantly rough patches. > (Insert standard "good documentation is hard and often boring work, and > anyone who volunteers to write accurate documentation that's even > vaguely legible is a minor hero" disclaimer here.) Contributions are welcome ;-) (I do not remember how many times I have tried to rewrite that particular paragraph) -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From vans_163@REDACTED Thu Jan 19 17:01:09 2017 From: vans_163@REDACTED (Vans S) Date: Thu, 19 Jan 2017 16:01:09 +0000 (UTC) Subject: [erlang-questions] gen_statem confusion In-Reply-To: <20170119101348.GA36166@erix.ericsson.se> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> <20170119002742.GD26705@ferdmbp.local> <634078528.341606.1484794867558@mail.yahoo.com> <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> <20170119101348.GA36166@erix.ericsson.se> Message-ID: <924714402.803002.1484841669047@mail.yahoo.com> > So if you in a state callback insert more next_event messages these are > inserted first in the queue, and a state_timeout 0 message is inserted last > in the queue. This clarifies things nicely. Also what if one was to use {next_event, state_timeout .. vs {state_timeout .. Also is there a priority such as external, internal, timeout, state_timeout, etc? Or anything with next_event goes first into the queue (or top of stack if using {next_event, external .. ?) regardless queue or stack the idea is the same, it will be first to dequeue/pop. And this will happen before receive is called. On Thursday, January 19, 2017 5:14 AM, Raimo Niskanen wrote: On Thu, Jan 19, 2017 at 11:59:37AM +0300, Alex S. wrote: > > > 19 ???. 2017 ?., ? 6:01, Vans S ???????(?): > > > > > > > > This wording is really confusing: > > > >> instead the the time-out event is enqueued to ensure > >> that it gets processed before any not yet received > >> external event > > Because when I tried a test case > > > > init() -> > > send(self(), hi), > > {_,_,_,0}. %0 timeout basically > > > > The 0 timeout procced before the 'hi' message. > > Using 1 as the timeout, 'hi' message procs first. > > > > But it says not yet received external event. To me send() seems like > > an external event. Hence confusing. > Well, it was sent, but not yet received by the gen_statem?s main loop. Perfectly correct. While gen_statem processes enqueued events it is guaranteed to not enter a receive statement, so no new messages are received until after ther state_timeout 0 message. Other enqueued messages that are being processed such as next_event messages will be processed before a state_timeout 0 message. So if you in a state callback insert more next_event messages these are inserted first in the queue, and a state_timeout 0 message is inserted last in the queue. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From vans_163@REDACTED Thu Jan 19 17:15:54 2017 From: vans_163@REDACTED (Vans S) Date: Thu, 19 Jan 2017 16:15:54 +0000 (UTC) Subject: [erlang-questions] gen_statem confusion In-Reply-To: <1716449697.822124.1484842106762@mail.yahoo.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> <20170119002742.GD26705@ferdmbp.local> <634078528.341606.1484794867558@mail.yahoo.com> <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> <20170119101348.GA36166@erix.ericsson.se> <924714402.803002.1484841669047@mail.yahoo.com> <1716449697.822124.1484842106762@mail.yahoo.com> Message-ID: <591367985.818569.1484842554304@mail.yahoo.com> Yea ignore that. That is because I was not up to R19.2 when I tested. Let me get up to date. On Thursday, January 19, 2017 11:08 AM, Vans S wrote: Actually I jumped the gun here: using a state_timeout as next event which is a valid event_type() as said in the docs produces: ** Reason for termination = error:{bad_action_from_state_function, {next_event,state_timeout,hi}} So state_timeout cant be used here. timeout, cast, info can be used. {call, pid()} also cant be used. On Thursday, January 19, 2017 11:01 AM, Vans S wrote: > So if you in a state callback insert more next_event messages these are > inserted first in the queue, and a state_timeout 0 message is inserted last > in the queue. This clarifies things nicely. Also what if one was to use {next_event, state_timeout .. vs {state_timeout .. Also is there a priority such as external, internal, timeout, state_timeout, etc? Or anything with next_event goes first into the queue (or top of stack if using {next_event, external .. ?) regardless queue or stack the idea is the same, it will be first to dequeue/pop. And this will happen before receive is called. On Thursday, January 19, 2017 5:14 AM, Raimo Niskanen wrote: On Thu, Jan 19, 2017 at 11:59:37AM +0300, Alex S. wrote: > > > 19 ???. 2017 ?., ? 6:01, Vans S ???????(?): > > > > > > > > This wording is really confusing: > > > >> instead the the time-out event is enqueued to ensure > >> that it gets processed before any not yet received > >> external event > > Because when I tried a test case > > > > init() -> > > send(self(), hi), > > {_,_,_,0}. %0 timeout basically > > > > The 0 timeout procced before the 'hi' message. > > Using 1 as the timeout, 'hi' message procs first. > > > > But it says not yet received external event. To me send() seems like > > an external event. Hence confusing. > Well, it was sent, but not yet received by the gen_statem?s main loop. Perfectly correct. While gen_statem processes enqueued events it is guaranteed to not enter a receive statement, so no new messages are received until after ther state_timeout 0 message. Other enqueued messages that are being processed such as next_event messages will be processed before a state_timeout 0 message. So if you in a state callback insert more next_event messages these are inserted first in the queue, and a state_timeout 0 message is inserted last in the queue. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From vans_163@REDACTED Thu Jan 19 17:35:19 2017 From: vans_163@REDACTED (Vans S) Date: Thu, 19 Jan 2017 16:35:19 +0000 (UTC) Subject: [erlang-questions] gen_statem confusion In-Reply-To: <591367985.818569.1484842554304@mail.yahoo.com> References: <1798520640.8723249.1484764836743.ref@mail.yahoo.com> <1798520640.8723249.1484764836743@mail.yahoo.com> <20170118192024.GG18549@fhebert-ltm2.internal.salesforce.com> <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> <20170119002742.GD26705@ferdmbp.local> <634078528.341606.1484794867558@mail.yahoo.com> <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> <20170119101348.GA36166@erix.ericsson.se> <924714402.803002.1484841669047@mail.yahoo.com> <1716449697.822124.1484842106762@mail.yahoo.com> <591367985.818569.1484842554304@mail.yahoo.com> Message-ID: <1121688889.823044.1484843719640@mail.yahoo.com> Okay same problem with R19.2.1 On Thursday, January 19, 2017 11:15 AM, Vans S wrote: Yea ignore that. That is because I was not up to R19.2 when I tested. Let me get up to date. On Thursday, January 19, 2017 11:08 AM, Vans S wrote: Actually I jumped the gun here: using a state_timeout as next event which is a valid event_type() as said in the docs produces: ** Reason for termination = error:{bad_action_from_state_function, {next_event,state_timeout,hi}} So state_timeout cant be used here. timeout, cast, info can be used. {call, pid()} also cant be used. On Thursday, January 19, 2017 11:01 AM, Vans S wrote: > So if you in a state callback insert more next_event messages these are > inserted first in the queue, and a state_timeout 0 message is inserted last > in the queue. This clarifies things nicely. Also what if one was to use {next_event, state_timeout .. vs {state_timeout .. Also is there a priority such as external, internal, timeout, state_timeout, etc? Or anything with next_event goes first into the queue (or top of stack if using {next_event, external .. ?) regardless queue or stack the idea is the same, it will be first to dequeue/pop. And this will happen before receive is called. On Thursday, January 19, 2017 5:14 AM, Raimo Niskanen wrote: On Thu, Jan 19, 2017 at 11:59:37AM +0300, Alex S. wrote: > > > 19 ???. 2017 ?., ? 6:01, Vans S ???????(?): > > > > > > > > This wording is really confusing: > > > >> instead the the time-out event is enqueued to ensure > >> that it gets processed before any not yet received > >> external event > > Because when I tried a test case > > > > init() -> > > send(self(), hi), > > {_,_,_,0}. %0 timeout basically > > > > The 0 timeout procced before the 'hi' message. > > Using 1 as the timeout, 'hi' message procs first. > > > > But it says not yet received external event. To me send() seems like > > an external event. Hence confusing. > Well, it was sent, but not yet received by the gen_statem?s main loop. Perfectly correct. While gen_statem processes enqueued events it is guaranteed to not enter a receive statement, so no new messages are received until after ther state_timeout 0 message. Other enqueued messages that are being processed such as next_event messages will be processed before a state_timeout 0 message. So if you in a state callback insert more next_event messages these are inserted first in the queue, and a state_timeout 0 message is inserted last in the queue. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Thu Jan 19 19:00:35 2017 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 19 Jan 2017 10:00:35 -0800 Subject: [erlang-questions] A math module better than C89 In-Reply-To: <5d72f814-7dc6-bb56-96d8-fc68035ebbc8@cs.otago.ac.nz> References: <58805B64.6080704@gmail.com> <5d72f814-7dc6-bb56-96d8-fc68035ebbc8@cs.otago.ac.nz> Message-ID: <5880FEC3.5040102@gmail.com> On 01/19/2017 12:28 AM, Richard A. O'Keefe wrote: > I honestly don't think there's much difficulty figuring > out what the math functions *should* do. > > To start with, we cannot offer a math library better than > C89 until we offer a math library as good as C89, and we > just don't do that. > - there is no frexp and that turns out to matter a lot > - there is no ldexp > - there is no floor (round is provided elsewhere) > - there is no ceil (trunc is provided elsewhere) > - there is no fmod > - there is no modf > There's also a function that was in the Unix libraries but > didn't make it into the C standard until 1999: > - there is no hypot (hypot and atan2 are *partners*). > > Since it's now 18 years since C99, shouldn't we be thinking > about a math module better than C99? (The real part anyway.) > > Let's review what's in the math module, working from > http://erlang.org/doc/man/math.html > > In working on my Smalltalk library, I wrote support > functions to convert a bignum to a normalised > fraction in float, double, or long double form and > an integer power of 2, rather like frexp. Putting > this in Erlang terms, if we have > frexp(Number) -> {Fraction, Scale} > then > log(Number) -> > {Fraction, Scale} = frexp(Number), > underlying:log(Fraction) + Scale * underlying:log(2.0). > > This means not having to worry about floating point overflow > when dealing with logs of (positive) bignums. > > As it happens, ANSI Smalltalk does not include a logarithm > operation on integers, so I *didn't* do this. Before going > to sleep tonight, I will! > > acos(X) > X must be between -1 and 1 inclusive. > bignums must be reported as wrong. > acosh(X) > log(X + (X+1)sqrt((X-1)/(X+1)) > (X-1)/(X+1) will be effectively 1, > sqrt((X-1)/(X+1)) will be effectively 1, > acosh(X) will be approximately log(2X+1). > See note about log above. > > asin(X) > X must be between -1 and 1 inclusive. > bignums must be reported as wrong. > asinh(X) > log(X + sqrt(1 + X**2)) > X**2 + 1 will be effectively 1, > sqrt(X**2 + 1) will be effectively |X|, > asinh(X) will be effectively log(2X). > See note about log above. > > atan(X) > Bignums that would be converted to ?inf > may as well be allowed to, > atan(?inf) = ?pi > > atan2(X, Y) > After dealing with X == 0 or Y == 0 specially, > remember the signs. We can work with > {FX, SX} = frexp(X), > {FY, SY} = frexp(Y), > FR = FY/FX, SR = SY-SX > R = if SR < lower threshold -> 0 > ; SR > upper threshold -> +infinity > ; true -> ldexp(FR, SR) > end > Now compute the result from R and the saved signs. > > atanh(X) > tanh(Y) lies between -1 and 1 inclusive, so > any integer, big or small, other than -1, 0, 1 > is an error. > > MISSING: cbrt/1. > See sqrt. > > cos(X) > To be honest, I don't think this really makes sense. > I know that the Sun math library claimed to get as > good an answer as you could hope for using > "infinitely precise pi". I have no idea how that > works. The old UNIX libm interface used to report > doubles that were too big as basically not terribly > meaningful. (If an ulp is larger than pi, you > really cannot represent an angle in any meaningful > way. And that happens about 2.8e16. So frankly, > if a bignum would convert to infinity, I say let > it, and let the C library tell you that cos(inf) is NaN. > cosh(X) > (exp(X) + exp(-X))/2 will be near exp(|X|). > +infinity for a bignum. See exp. > > exp(X) > The result is a float. For any float X greater than > about 709.78 the answer will be +infinity. > A negative bignum can give the answer 0.0. > A positive bignum should give the answer +infinity. > > MISSING AND IMPORTANT: expm1/1 > > log(X) > Described above. We can do this! > > log10(X) > is log(X)/log(10) > > log2(X) > is log(X)/log(2) > > MISSING AND IMPORTANT: log1p/1 > > pow(X, Y) > Smalltalk distinguishes between number raisedToInteger: integer > (which typically uses the bit oriented square and multiply > approach) and base raisedTo: power which reverts to > raisedToInteger: if power is an integer otherwise uses > exp(log(base) * power). > This needs a careful case analysis. > pow(Bignum, Float) will of course only be allowed > when Bignum > 0. Since we can compute log(Bignum), > this reduces to exp(log(Bignum)*Float). Sometimes > that will give 0, sometimes a finite number, and > sometimes ?infinity. > It's all doable by careful case analysis. > > sin(X) > See comment on cos(X). > sinh(X) > (exp(X) - exp(-X))/2. > If bignum X > 0, -> exp(X)/2 -> +infinity. > If bignum X < 0, -> -exp(-X)/2 -> -infinity. > > sqrt(X) > Smalltalk actually defines sqrt on integers as integer > square root, which is a pain in the posterior. I have > wanted the integer square root function occasionally, > true. But I have much more often wanted > anInteger sqrt -> anInteger asFloat sqrt, which is what > Erlang does, hooray, hooray. > If bignum X < 0, this is an error. > Let {F, S} = frexp(X). > Let {G, T} = if S odd -> {sqrt(F*2.0), (S-1) div 2} > ; S even -> (sqrt(F), S div 2) > end. > Answer ldexp(G, T). > This will give you good answers up to 3.2x10**616 > and then infinity. > > tan(X) > See comment on cos. > tanh(X) > (exp(X) - exp(-X))/(exp(X) + exp(-X)) > For any X > about 709.78, answer +infinity. > For any X < about -709.78, answer -infinity. > > erf(X) > From the graph of erf at http://mathworld.wolfram.com/Erf.html > it is pretty obvious that > For bignum X > 0, erf(X) -> 1.0 and erfc(X) -> 0.0 > For bignum X < 0, erf(X) -> -1.0 and erfc(X) -> 2.0 > > pi() > has no argument, so no bignum issue. > > > I presume pow(X, Y, M) computes (X mod M)**Y mod M? > This reduces to multiplication modulo M > (is that '*'(X, Y, M) ?) which is not problematic. > > The core here is two operations that have been in the C library > (and in many languages under various guises) but are not in the > Erlang library: frexp and ldexp. I have a faint recollection > of libgmp offering frexp, if not I could donate my C code. > > Of course many of these things are in other languages with > bignums. (Hands up everyone who realised I was staring at > page 308 of CLtL2? Which doesn't actually mention bignums.) > > By the way, my HoD and HR are growling at me for not publishing > enough, and a friend this afternoon said "don't be ashamed of > publishing simple things". Could I get a paper out of this > if I finished the case analyses and wrapped a few more words > around it? (Seriously, I am quite stressed about the situation.) > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions We do have floor/ceil now (from https://github.com/erlang/otp/pull/1145) when the source code is released in Erlang/OTP 20.0 and these functions are guard functions, though that means they both exist in the erlang module, not the math module. |crypto:mod_pow/3 exists, but the math module would want its own implementation to avoid a dependency on crypto. libgmp has pow functions for bigints though the situation gets more complex when handling floating point usage with bigints, as you have described. Thank you for identifying the missing pieces and the specific problems! | -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Jan 20 00:30:26 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 20 Jan 2017 12:30:26 +1300 Subject: [erlang-questions] A math module better than C89 In-Reply-To: <5880FEC3.5040102@gmail.com> References: <58805B64.6080704@gmail.com> <5d72f814-7dc6-bb56-96d8-fc68035ebbc8@cs.otago.ac.nz> <5880FEC3.5040102@gmail.com> Message-ID: <25a0b4f8-b1b6-a902-9b9f-bcdedd5c1435@cs.otago.ac.nz> On 20/01/17 7:00 AM, Michael Truog wrote: > We do have floor/ceil now (from https://github.com/erlang/otp/pull/1145) > when the source code is released in Erlang/OTP 20.0 and these functions > are guard functions, though that means they both exist in the erlang > module, not the math module. That is good to hear. As a minor documentation issue, because there will be people exposed to C or Java who expect functions called round, trunc, floor, ceil to be in the same place as sqrt and acosh, it would be helpful if the manual page for the math module mentioned these functions, maybe at the end, and said where to really find them. I'm not talking about repeating their descriptions, just pointing to them. From mjtruog@REDACTED Fri Jan 20 01:02:58 2017 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 19 Jan 2017 16:02:58 -0800 Subject: [erlang-questions] A math module better than C89 In-Reply-To: <25a0b4f8-b1b6-a902-9b9f-bcdedd5c1435@cs.otago.ac.nz> References: <58805B64.6080704@gmail.com> <5d72f814-7dc6-bb56-96d8-fc68035ebbc8@cs.otago.ac.nz> <5880FEC3.5040102@gmail.com> <25a0b4f8-b1b6-a902-9b9f-bcdedd5c1435@cs.otago.ac.nz> Message-ID: <588153B2.4030302@gmail.com> On 01/19/2017 03:30 PM, Richard A. O'Keefe wrote: > > > On 20/01/17 7:00 AM, Michael Truog wrote: > >> We do have floor/ceil now (from https://github.com/erlang/otp/pull/1145) >> when the source code is released in Erlang/OTP 20.0 and these functions >> are guard functions, though that means they both exist in the erlang >> module, not the math module. > > That is good to hear. > > As a minor documentation issue, because there will be people exposed to > C or Java who expect functions called round, trunc, floor, ceil to be > in the same place as sqrt and acosh, it would be helpful if the > manual page for the math module mentioned these functions, maybe at the > end, and said where to really find them. I'm not talking about > repeating their descriptions, just pointing to them. I was wrong about this a bit. There are ceil/floor functions added to the math function that use the same logic as the erlang module ceil/floor functions. The main difference is that the math module return type is float while the erlang module return type is integer. So, ceil/floor are present in the math module, though the math module doc probably wants a pointer to the erlang module ceil/floor by mentioning that it returns integer instead of float. From raimo+erlang-questions@REDACTED Fri Jan 20 09:00:59 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 20 Jan 2017 09:00:59 +0100 Subject: [erlang-questions] gen_statem confusion In-Reply-To: <1121688889.823044.1484843719640@mail.yahoo.com> References: <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> <20170119002742.GD26705@ferdmbp.local> <634078528.341606.1484794867558@mail.yahoo.com> <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> <20170119101348.GA36166@erix.ericsson.se> <924714402.803002.1484841669047@mail.yahoo.com> <1716449697.822124.1484842106762@mail.yahoo.com> <591367985.818569.1484842554304@mail.yahoo.com> <1121688889.823044.1484843719640@mail.yahoo.com> Message-ID: <20170120080059.GA97445@erix.ericsson.se> On Thu, Jan 19, 2017 at 04:35:19PM +0000, Vans S wrote: > Okay same problem with R19.2.1 > > > > On Thursday, January 19, 2017 11:15 AM, Vans S wrote: > Yea ignore that. That is because I was not up to R19.2 when I tested. > Let me get up to date. > > > > > On Thursday, January 19, 2017 11:08 AM, Vans S wrote: > Actually I jumped the gun here: > > > using a state_timeout as next event which is a valid > event_type() as said in the docs produces: > > ** Reason for termination = error:{bad_action_from_state_function, > {next_event,state_timeout,hi}} > So state_timeout cant be used here. > > timeout, cast, info can be used. > > {call, pid()} also cant be used. > That is a bug! I missed to add a clause in event_type/1 for state_timeout when I introduced it. The missing clause for {call,From} has always not been there - amazing nobody noticed! I guess generating events of type 'internal' is the really useful feature - the other types are mostly of theoretical value. Using them will most probably be confusing. I will try to squeeze in to fix that for 19.3. > > > > On Thursday, January 19, 2017 11:01 AM, Vans S wrote: > > So if you in a state callback insert more next_event messages these are > > inserted first in the queue, and a state_timeout 0 message is inserted last > > in the queue. > > This clarifies things nicely. Also what if one was to use {next_event, state_timeout .. vs {state_timeout .. > Also is there a priority such as external, internal, timeout, state_timeout, etc? There is no priority between types. First to enqueue is first to dequeue, except for {next_event,_,_} actions that will be _before_ first to dequeue. > > Or anything with next_event goes first into the queue (or top of stack if using {next_event, external .. ?) > regardless queue or stack the idea is the same, it will be first to dequeue/pop. And this will happen before receive > is called. Precisely. All {next_event,_,_} actions in an event handler return will be the first to dequeue for the next event handler, and the order between them is that the first in the event handler action list will be the first to dequeue. So when the bug above has been fixed an action {next_event,state_timeout,_} will produce an event that would be dequeued before the event from an action {state_timeout,0,_}, with possible already enqueued events in between. Furthermore multiple {next_event,state_timeout,_} actions will produce one event each, but multiple {state_timeout,0,_} actions will override each other so the last one wins hence only produce one event. > > > > > > On Thursday, January 19, 2017 5:14 AM, Raimo Niskanen wrote: > On Thu, Jan 19, 2017 at 11:59:37AM +0300, Alex S. wrote: > > > > > > 19 ???. 2017 ?., ? 6:01, Vans S ???????(?): > > > > > > > > > > > > This wording is really confusing: > > > > > >> instead the the time-out event is enqueued to ensure > > >> that it gets processed before any not yet received > > >> external event > > > Because when I tried a test case > > > > > > init() -> > > > send(self(), hi), > > > {_,_,_,0}. %0 timeout basically > > > > > > The 0 timeout procced before the 'hi' message. > > > Using 1 as the timeout, 'hi' message procs first. > > > > > > But it says not yet received external event. To me send() seems like > > > an external event. Hence confusing. > > Well, it was sent, but not yet received by the gen_statem?s main loop. > > Perfectly correct. > > While gen_statem processes enqueued events it is guaranteed to not enter a > receive statement, so no new messages are received until after ther > state_timeout 0 message. > > Other enqueued messages that are being processed such as next_event > messages will be processed before a state_timeout 0 message. > > So if you in a state callback insert more next_event messages these are > inserted first in the queue, and a state_timeout 0 message is inserted last > in the queue. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From vans_163@REDACTED Fri Jan 20 16:58:59 2017 From: vans_163@REDACTED (Vans S) Date: Fri, 20 Jan 2017 15:58:59 +0000 (UTC) Subject: [erlang-questions] gen_statem confusion In-Reply-To: <20170120080059.GA97445@erix.ericsson.se> References: <20170118204820.GH18549@fhebert-ltm2.internal.salesforce.com> <1858145896.95613.1484777025374@mail.yahoo.com> <20170119002742.GD26705@ferdmbp.local> <634078528.341606.1484794867558@mail.yahoo.com> <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> <20170119101348.GA36166@erix.ericsson.se> <924714402.803002.1484841669047@mail.yahoo.com> <1716449697.822124.1484842106762@mail.yahoo.com> <591367985.818569.1484842554304@mail.yahoo.com> <1121688889.823044.1484843719640@mail.yahoo.com> <20170120080059.GA97445@erix.ericsson.se> Message-ID: <2110543021.754700.1484927939504@mail.yahoo.com> > There is no priority between types. First to enqueue is first to dequeue, > except for {next_event,_,_} actions that will be _before_ first to dequeue. This is logical and simple to reason about. Nice. > So when the bug above has been fixed an action {next_event,state_timeout,_} > will produce an event that would be dequeued before the event from an action > {state_timeout,0,_}, with possible already enqueued events in between. > Furthermore multiple {next_event,state_timeout,_} actions will produce > one event each, but multiple {state_timeout,0,_} actions will override > each other so the last one wins hence only produce one event. Got it. One of my projects is itching to have state_timeout. Right now I am working with gen_statem to handle netsplits and I think there maybe a need to have a way for the gen_statem to not process any call/cast/info until it gets to a certain state. Take for example we have a need to implement pairs trading for stocks. Using states such as: step0_join_distribution step1_subscribe_to_ticker1 step2_subscribe_to_ticker2 step3_process_messages Our tree looks like: nodedown -> {next_state, step0_join_distribution, {state_timeout, 10000, _}} step0_join_distribution - monitor_nodes_for_nodedown step1_subscribe_to_ticker1 - udp:subscribe - if subscribed -> monitor_for_sub_loss, step2_subscribe_to_ticker2 - else {state_timeout, 1000, _} step2_subscribe_to_ticker2 - udp:subscribe - if subscribed -> monitor_for_sub_loss, step3_process_messages - else {state_timeout, 1000, _} step3_process_messages - Now we can process messages normally that come from - other parts of the distribution. The issue I am grappling with now is if we fail on the subscription to the 3rd party API (subscription via udp or tcp) we need to try again. Trying again we use state_timeout but this opens a gap for messages to be processed. A possible approach would be to have a catch_all handler for cast/call/info for all states that =/= step3_process_messages, it can offload the mailbox to state for later processing or reply with a certain error like not_ready_for_msgs. But yea just some thoughts on use cases. On Friday, January 20, 2017 3:03 AM, Raimo Niskanen wrote: On Thu, Jan 19, 2017 at 04:35:19PM +0000, Vans S wrote: > Okay same problem with R19.2.1 > > > > On Thursday, January 19, 2017 11:15 AM, Vans S wrote: > Yea ignore that. That is because I was not up to R19.2 when I tested. > Let me get up to date. > > > > > On Thursday, January 19, 2017 11:08 AM, Vans S wrote: > Actually I jumped the gun here: > > > using a state_timeout as next event which is a valid > event_type() as said in the docs produces: > > ** Reason for termination = error:{bad_action_from_state_function, > {next_event,state_timeout,hi}} > So state_timeout cant be used here. > > timeout, cast, info can be used. > > {call, pid()} also cant be used. > That is a bug! I missed to add a clause in event_type/1 for state_timeout when I introduced it. The missing clause for {call,From} has always not been there - amazing nobody noticed! So when the bug above has been fixed an action {next_event,state_timeout,_} will produce an event that would be dequeued before the event from an action {state_timeout,0,_}, with possible already enqueued events in between. Furthermore multiple {next_event,state_timeout,_} actions will produce one event each, but multiple {state_timeout,0,_} actions will override each other so the last one wins hence only produce one event. I guess generating events of type 'internal' is the really useful feature - the other types are mostly of theoretical value. Using them will most probably be confusing. I will try to squeeze in to fix that for 19.3. > > > > On Thursday, January 19, 2017 11:01 AM, Vans S wrote: > > So if you in a state callback insert more next_event messages these are > > inserted first in the queue, and a state_timeout 0 message is inserted last > > in the queue. > > This clarifies things nicely. Also what if one was to use {next_event, state_timeout .. vs {state_timeout .. > Also is there a priority such as external, internal, timeout, state_timeout, etc? There is no priority between types. First to enqueue is first to dequeue, except for {next_event,_,_} actions that will be _before_ first to dequeue. > > Or anything with next_event goes first into the queue (or top of stack if using {next_event, external .. ?) > regardless queue or stack the idea is the same, it will be first to dequeue/pop. And this will happen before receive > is called. Precisely. All {next_event,_,_} actions in an event handler return will be the first to dequeue for the next event handler, and the order between them is that the first in the event handler action list will be the first to dequeue. So when the bug above has been fixed an action {next_event,state_timeout,_} will produce an event that would be dequeued before the event from an action {state_timeout,0,_}, with possible already enqueued events in between. Furthermore multiple {next_event,state_timeout,_} actions will produce one event each, but multiple {state_timeout,0,_} actions will override each other so the last one wins hence only produce one event. > > > > > > On Thursday, January 19, 2017 5:14 AM, Raimo Niskanen wrote: > On Thu, Jan 19, 2017 at 11:59:37AM +0300, Alex S. wrote: > > > > > > 19 ???. 2017 ?., ? 6:01, Vans S ???????(?): > > > > > > > > > > > > This wording is really confusing: > > > > > >> instead the the time-out event is enqueued to ensure > > >> that it gets processed before any not yet received > > >> external event > > > Because when I tried a test case > > > > > > init() -> > > > send(self(), hi), > > > {_,_,_,0}. %0 timeout basically > > > > > > The 0 timeout procced before the 'hi' message. > > > Using 1 as the timeout, 'hi' message procs first. > > > > > > But it says not yet received external event. To me send() seems like > > > an external event. Hence confusing. > > Well, it was sent, but not yet received by the gen_statem?s main loop. > > Perfectly correct. > > While gen_statem processes enqueued events it is guaranteed to not enter a > receive statement, so no new messages are received until after ther > state_timeout 0 message. > > Other enqueued messages that are being processed such as next_event > messages will be processed before a state_timeout 0 message. > > So if you in a state callback insert more next_event messages these are > inserted first in the queue, and a state_timeout 0 message is inserted last > in the queue. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-question From mononcqc@REDACTED Fri Jan 20 17:12:52 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 20 Jan 2017 11:12:52 -0500 Subject: [erlang-questions] gen_statem confusion In-Reply-To: <2110543021.754700.1484927939504@mail.yahoo.com> References: <20170119002742.GD26705@ferdmbp.local> <634078528.341606.1484794867558@mail.yahoo.com> <5ED7B915-476D-4CA2-80FD-6E9CADD2BE83@gmail.com> <20170119101348.GA36166@erix.ericsson.se> <924714402.803002.1484841669047@mail.yahoo.com> <1716449697.822124.1484842106762@mail.yahoo.com> <591367985.818569.1484842554304@mail.yahoo.com> <1121688889.823044.1484843719640@mail.yahoo.com> <20170120080059.GA97445@erix.ericsson.se> <2110543021.754700.1484927939504@mail.yahoo.com> Message-ID: <20170120161250.GI18549@fhebert-ltm2.internal.salesforce.com> On 01/20, Vans S wrote: > >The issue I am grappling with now is if we fail on the subscription to the 3rd party API >(subscription via udp or tcp) we need to try again. Trying again we use state_timeout >but this opens a gap for messages to be processed. A possible approach would be to have a >catch_all handler for cast/call/info for all states that =/= step3_process_messages, it can >offload the mailbox to state for later processing or reply with a certain error like >not_ready_for_msgs. > You may only need to use `postpone` functionality, which will re-enqueue the message until a state change. From sverker.eriksson@REDACTED Fri Jan 20 17:17:41 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 20 Jan 2017 17:17:41 +0100 Subject: [erlang-questions] Problem with binary_to_term on different machines In-Reply-To: References: Message-ID: <1fc4b57e-17ba-0bd5-7f1b-e451cf34f10a@ericsson.com> This is indeed a problem in Erlang VM code (shallow copy of inflate state) that has existed since R16B03, but not caused actual problem until zlib v1.2.9. Fix coming up. Here is a preliminary patch for the impatient. diff --git a/erts/emulator/beam/external.c b/erts/emulator/beam/external.c index beed847..1c4fff5 100644 --- a/erts/emulator/beam/external.c +++ b/erts/emulator/beam/external.c @@ -1431,6 +1431,10 @@ static B2TContext* b2t_export_context(Process* p, B2TContext* src) if (ctx->state >= B2TDecode && ctx->u.dc.next == &src->u.dc.res) { ctx->u.dc.next = &ctx->u.dc.res; } + else if (ctx->state == B2TUncompressChunk) { + int cres = inflateCopy(&ctx->u.uc.stream, &src->u.uc.stream); + ASSERT(cres == Z_OK); (void)cres; + } hp = HAlloc(p, PROC_BIN_SIZE); ctx->trap_bin = erts_mk_magic_binary_term(&hp, &MSO(p), context_b); return ctx; /Sverker, Erlang/OTP On 01/19/2017 04:14 AM, Michel Boaventura wrote: > Hi all, > > I've build erlang 19 from scratch on my gentoo machine but can't use > dialyzer, because it complains my beam files weren't compiled with > debug_info. > > I've read and debugged a log of dialyzer and erlang code and find out the > issue is that for some reason erlang can't parse the abstract part of the > files. > > To check this I save the abstract part on a file and tried to parse it on > my gentoo machine and on a Ubuntu server I have access. For some reason I > don't know it reads with no problem Ubuntu but gives me an invalid arg on > gentoo. > > I'm very new to erlang and have no clue of how to debug further. The binary > is attached and I'm reading it like: > > {ok, [Binary]} = file:consult("Abstr"). > Term = binary_to_term(Binary). > > ps: I've sent a message similar to this one on this list, but since it was > my first one I think it wasn't accepted (and I do apologize for the double > post if you received it). But after it I've discovered the problem with > binary_to_term, so that message isn't so useful anyway. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Fri Jan 20 17:45:26 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 20 Jan 2017 17:45:26 +0100 Subject: [erlang-questions] Problem with binary_to_term on different machines In-Reply-To: <1fc4b57e-17ba-0bd5-7f1b-e451cf34f10a@ericsson.com> References: <1fc4b57e-17ba-0bd5-7f1b-e451cf34f10a@ericsson.com> Message-ID: Correction: Bug exists since OTP-17.0. (and i tags R16B02_yielding_binary_to_term and OTP_R16B03_yielding_binary_to_term) /Sverker On 01/20/2017 05:17 PM, Sverker Eriksson wrote: > This is indeed a problem in Erlang VM code (shallow copy of inflate > state) > that has existed since R16B03, but not caused actual problem until > zlib v1.2.9. > > Fix coming up. Here is a preliminary patch for the impatient. > > diff --git a/erts/emulator/beam/external.c > b/erts/emulator/beam/external.c > index beed847..1c4fff5 100644 > --- a/erts/emulator/beam/external.c > +++ b/erts/emulator/beam/external.c > @@ -1431,6 +1431,10 @@ static B2TContext* b2t_export_context(Process* > p, B2TContext* src) > if (ctx->state >= B2TDecode && ctx->u.dc.next == &src->u.dc.res) { > ctx->u.dc.next = &ctx->u.dc.res; > } > + else if (ctx->state == B2TUncompressChunk) { > + int cres = inflateCopy(&ctx->u.uc.stream, &src->u.uc.stream); > + ASSERT(cres == Z_OK); (void)cres; > + } > hp = HAlloc(p, PROC_BIN_SIZE); > ctx->trap_bin = erts_mk_magic_binary_term(&hp, &MSO(p), context_b); > return ctx; > > > /Sverker, Erlang/OTP > > > On 01/19/2017 04:14 AM, Michel Boaventura wrote: >> Hi all, >> >> I've build erlang 19 from scratch on my gentoo machine but can't use >> dialyzer, because it complains my beam files weren't compiled with >> debug_info. >> >> I've read and debugged a log of dialyzer and erlang code and find out >> the >> issue is that for some reason erlang can't parse the abstract part of >> the >> files. >> >> To check this I save the abstract part on a file and tried to parse >> it on >> my gentoo machine and on a Ubuntu server I have access. For some >> reason I >> don't know it reads with no problem Ubuntu but gives me an invalid >> arg on >> gentoo. >> >> I'm very new to erlang and have no clue of how to debug further. The >> binary >> is attached and I'm reading it like: >> >> {ok, [Binary]} = file:consult("Abstr"). >> Term = binary_to_term(Binary). >> >> ps: I've sent a message similar to this one on this list, but since >> it was >> my first one I think it wasn't accepted (and I do apologize for the >> double >> post if you received it). But after it I've discovered the problem with >> binary_to_term, so that message isn't so useful anyway. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From petergi@REDACTED Sat Jan 21 03:53:48 2017 From: petergi@REDACTED (Peter J Etheridge) Date: Sat, 21 Jan 2017 13:53:48 +1100 Subject: [erlang-questions] math module Message-ID: <8a80482f3cb1687c8273f3ef9c9b6f29d24d0190@webmail.iinet.net.au> On?19 Jan 2017? "Richard A. O'Keefe" In 'A math module better than C89',?ROK revealed? ...'my HoD and HR are growling at me for not publishing?enough...' ROK's contribution on math module alone, along with; 'A compilation of answers to Erlang questions byRichard A. O'Keefe' would make a worthy publication indeed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuna.prime@REDACTED Sat Jan 21 13:27:09 2017 From: kuna.prime@REDACTED (Karlo Kuna) Date: Sat, 21 Jan 2017 13:27:09 +0100 Subject: [erlang-questions] Implementation of erlang binary module Message-ID: I was wandering how is standard module "binary" implemented i am looking at source and i see that most functions are nifs (this makes sense), but there is no "-on_load" directive and i can't find c implementation. Please can someone explain to me how is this module using nifs without -on_load directive and also can tell me where is c implementation?? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Sat Jan 21 14:38:12 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Sat, 21 Jan 2017 14:38:12 +0100 Subject: [erlang-questions] Implementation of erlang binary module In-Reply-To: References: Message-ID: Standard library rarely uses dynamic loaded SO/DLL modules (so called NIFs), rather they use BIF functions, code for them is included straight into Erlang/OTP C source and follows different rules than NIF. Have a search through source C files for binary__, with high chance you will find the C source by this name. 2017-01-21 13:27 GMT+01:00 Karlo Kuna : > I was wandering how is standard module "binary" implemented > i am looking at source and i see that most functions are nifs (this makes > sense), > but there is no "-on_load" directive and i can't find c implementation. > Please can someone explain to me how is this module using nifs without > -on_load directive and also can tell me where is c implementation?? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuna.prime@REDACTED Sat Jan 21 15:15:25 2017 From: kuna.prime@REDACTED (Karlo Kuna) Date: Sat, 21 Jan 2017 15:15:25 +0100 Subject: [erlang-questions] Implementation of erlang binary module In-Reply-To: References: Message-ID: would this imply that erlang runtime is choosing to call bif if present instead of erlang implemented function ??? is this general case or these "alternative" implementations should be registered somehow somewhere?? and thak you for the search tip it worked On Sat, Jan 21, 2017 at 2:38 PM, Dmytro Lytovchenko < dmytro.lytovchenko@REDACTED> wrote: > Standard library rarely uses dynamic loaded SO/DLL modules (so called > NIFs), rather they use BIF functions, code for them is included straight > into Erlang/OTP C source and follows different rules than NIF. Have a > search through source C files for binary__, with high > chance you will find the C source by this name. > > 2017-01-21 13:27 GMT+01:00 Karlo Kuna : > >> I was wandering how is standard module "binary" implemented >> i am looking at source and i see that most functions are nifs (this makes >> sense), >> but there is no "-on_load" directive and i can't find c implementation. >> Please can someone explain to me how is this module using nifs without >> -on_load directive and also can tell me where is c implementation?? >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Sat Jan 21 16:48:29 2017 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sat, 21 Jan 2017 16:48:29 +0100 Subject: [erlang-questions] wildcarded resource counters in Gproc In-Reply-To: References: Message-ID: <8A3DC2E3-4584-4949-999E-A37E42399674@gmail.com> Out of curiosity, shouldn't you use ets:update_counter/4? From zzantozz@REDACTED Sat Jan 21 17:07:17 2017 From: zzantozz@REDACTED (Ryan Stewart) Date: Sat, 21 Jan 2017 16:07:17 +0000 Subject: [erlang-questions] Erlang 19 callbacks aren't attributes? In-Reply-To: References: Message-ID: Anybody? This seems like a moderately serious issue. We've found that it can cause issues with meck, which assumes callbacks appear in module_info attributes. On Mon, Jan 16, 2017 at 1:08 PM Ryan Stewart wrote: > There was a question about this on the list a few months back, but it was > unanswered. It seems that in Erlang 19, a module's callbacks no longer > appear in the module_info() as 'attributes'. They are, however, included in > the abstract code (via beam_lib). I'm wondering if this was an intentional > change or an oversight. > > Are callbacks supposed to show up under a module's module_info attributes > like they used to? > > Ryan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.boaventura@REDACTED Sat Jan 21 20:57:46 2017 From: michel.boaventura@REDACTED (Michel Boaventura) Date: Sat, 21 Jan 2017 17:57:46 -0200 Subject: [erlang-questions] Problem with binary_to_term on different machines In-Reply-To: References: <1fc4b57e-17ba-0bd5-7f1b-e451cf34f10a@ericsson.com> Message-ID: Thank you Sverker! On 20 January 2017 at 14:45, Sverker Eriksson wrote: > Correction: Bug exists since OTP-17.0. > > (and i tags R16B02_yielding_binary_to_term and > OTP_R16B03_yielding_binary_to_term) > > /Sverker > > On 01/20/2017 05:17 PM, Sverker Eriksson wrote: > > This is indeed a problem in Erlang VM code (shallow copy of inflate state) > that has existed since R16B03, but not caused actual problem until zlib > v1.2.9. > > Fix coming up. Here is a preliminary patch for the impatient. > > diff --git a/erts/emulator/beam/external.c b/erts/emulator/beam/external.c > > index beed847..1c4fff5 100644 > --- a/erts/emulator/beam/external.c > +++ b/erts/emulator/beam/external.c > @@ -1431,6 +1431,10 @@ static B2TContext* b2t_export_context(Process* p, > B2TContext* src) > if (ctx->state >= B2TDecode && ctx->u.dc.next == &src->u.dc.res) { > ctx->u.dc.next = &ctx->u.dc.res; > } > + else if (ctx->state == B2TUncompressChunk) { > + int cres = inflateCopy(&ctx->u.uc.stream, &src->u.uc.stream); > + ASSERT(cres == Z_OK); (void)cres; > + } > hp = HAlloc(p, PROC_BIN_SIZE); > ctx->trap_bin = erts_mk_magic_binary_term(&hp, &MSO(p), context_b); > return ctx; > > > /Sverker, Erlang/OTP > > > On 01/19/2017 04:14 AM, Michel Boaventura wrote: > > Hi all, > > I've build erlang 19 from scratch on my gentoo machine but can't use > dialyzer, because it complains my beam files weren't compiled with > debug_info. > > I've read and debugged a log of dialyzer and erlang code and find out the > issue is that for some reason erlang can't parse the abstract part of the > files. > > To check this I save the abstract part on a file and tried to parse it on > my gentoo machine and on a Ubuntu server I have access. For some reason I > don't know it reads with no problem Ubuntu but gives me an invalid arg on > gentoo. > > I'm very new to erlang and have no clue of how to debug further. The > binary > is attached and I'm reading it like: > > {ok, [Binary]} = file:consult("Abstr"). > Term = binary_to_term(Binary). > > ps: I've sent a message similar to this one on this list, but since it was > my first one I think it wasn't accepted (and I do apologize for the double > post if you received it). But after it I've discovered the problem with > binary_to_term, so that message isn't so useful anyway. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Michel Almada de Castro Boaventura Analista de Sistemas Laborat?rio de Software Livre - LSL -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Sun Jan 22 13:39:20 2017 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Sun, 22 Jan 2017 13:39:20 +0100 Subject: [erlang-questions] Implementation of erlang binary module In-Reply-To: References: Message-ID: Bifs are registered in special text file located in erts/emulator/beam/bif.tab, based on it an internal C array is generated: 1. bif_table which is traversed in beam_emu.c on first start (after init_emulator label) and erts_export_put is called on each table entry to populate exports table. 2. hash table in nbif_table in hipe_bif0.c that is some HiPE magics i am not familiar with. During the load time in beam_load.c function final_touch, the erts_export_put will be called for every export and will return existing export pointing at BIF instead of adding Erlang function version. 2017-01-21 15:15 GMT+01:00 Karlo Kuna : > would this imply that erlang runtime is choosing to call bif if present > instead of erlang implemented function ??? is this general case or these > "alternative" implementations should be registered somehow somewhere?? > > and thak you for the search tip it worked > > On Sat, Jan 21, 2017 at 2:38 PM, Dmytro Lytovchenko < > dmytro.lytovchenko@REDACTED> wrote: > >> Standard library rarely uses dynamic loaded SO/DLL modules (so called >> NIFs), rather they use BIF functions, code for them is included straight >> into Erlang/OTP C source and follows different rules than NIF. Have a >> search through source C files for binary__, with high >> chance you will find the C source by this name. >> >> 2017-01-21 13:27 GMT+01:00 Karlo Kuna : >> >>> I was wandering how is standard module "binary" implemented >>> i am looking at source and i see that most functions are nifs (this >>> makes sense), >>> but there is no "-on_load" directive and i can't find c implementation. >>> Please can someone explain to me how is this module using nifs without >>> -on_load directive and also can tell me where is c implementation?? >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuna.prime@REDACTED Sun Jan 22 13:44:41 2017 From: kuna.prime@REDACTED (Karlo Kuna) Date: Sun, 22 Jan 2017 13:44:41 +0100 Subject: [erlang-questions] Implementation of erlang binary module In-Reply-To: References: Message-ID: thak you for explanation, i will have look at source but now at least i have some orientation On Sun, Jan 22, 2017 at 1:39 PM, Dmytro Lytovchenko < dmytro.lytovchenko@REDACTED> wrote: > Bifs are registered in special text file located in > erts/emulator/beam/bif.tab, based on it an internal C array is generated: > > 1. bif_table which is traversed in beam_emu.c on first start (after > init_emulator label) and erts_export_put is called on each table entry to > populate exports table. > 2. hash table in nbif_table in hipe_bif0.c that is some HiPE magics i am > not familiar with. > > During the load time in beam_load.c function final_touch, the > erts_export_put will be called for every export and will return existing > export pointing at BIF instead of adding Erlang function version. > > 2017-01-21 15:15 GMT+01:00 Karlo Kuna : > >> would this imply that erlang runtime is choosing to call bif if present >> instead of erlang implemented function ??? is this general case or these >> "alternative" implementations should be registered somehow somewhere?? >> >> and thak you for the search tip it worked >> >> On Sat, Jan 21, 2017 at 2:38 PM, Dmytro Lytovchenko < >> dmytro.lytovchenko@REDACTED> wrote: >> >>> Standard library rarely uses dynamic loaded SO/DLL modules (so called >>> NIFs), rather they use BIF functions, code for them is included straight >>> into Erlang/OTP C source and follows different rules than NIF. Have a >>> search through source C files for binary__, with high >>> chance you will find the C source by this name. >>> >>> 2017-01-21 13:27 GMT+01:00 Karlo Kuna : >>> >>>> I was wandering how is standard module "binary" implemented >>>> i am looking at source and i see that most functions are nifs (this >>>> makes sense), >>>> but there is no "-on_load" directive and i can't find c implementation. >>>> Please can someone explain to me how is this module using nifs without >>>> -on_load directive and also can tell me where is c implementation?? >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thom.cherryhomes@REDACTED Sun Jan 22 21:56:12 2017 From: thom.cherryhomes@REDACTED (Thom Cherryhomes) Date: Sun, 22 Jan 2017 20:56:12 +0000 Subject: [erlang-questions] Complex state machines for complex telnet applications Message-ID: Hello, everyone. This is my first post to the list. I?m a long time swiss army knife of a software and hardware engineer who has started to pick up Erlang, because I believe it and languages like it to be the future of software development. With that said, I am working on a personal project of mine that I have wanted to complete, a 21st century version of a BBS system, and I believe Erlang and OTP are a good language and toolkit to write it in. Code is being dropped here: http://github.com/tschak909/nakbbs I am experimenting with both Ranch and just using gen_tcp with my own socket pools, but I do have some design questions. The telnet ports need to present a definite series of states, broken up into seperate pieces: I have definite common events that can be stretched across multiple use cases: * Display * Wait for line of input * Wait for character input * branch to another menu * hang up But there will also be definite sections of the system, which could be thought of as states: * Authentication * Message Browser * Message Editor * Various Games etc. and I?m trying to think of an appropriate architectural pattern that will avoid me basically having to cram every conceivable state as a sort of hierarchy of tuples into a single gen_statem module. How can I approach this in a nice abstract and extensible manner, that I could for example push into a Behaviour and subsequently utilize for different terminal types? It would be nice to split up the different parts of the system into applications that would essentially be called from well defined APIs? I?m just trying to figure this all out, and would love a semblence of direction so I can do the proper research and deeply understand what I need to learn. -Thom -------------- next part -------------- An HTML attachment was scrubbed... URL: From himars@REDACTED Mon Jan 23 08:36:48 2017 From: himars@REDACTED (Jack Tang) Date: Mon, 23 Jan 2017 15:36:48 +0800 Subject: [erlang-questions] How can I release beam process memory? In-Reply-To: References: Message-ID: Hello, In the memory issue investigation, I find something interesting. As I mentioned, we usually adopted remote Erlang shell to interactive with nodes in production. Sometime we use mnesia:select(...) to return some query result, and it may contain billions of rows. And we leave the remote shell alive. I find the query results do consume the memory of remote node, since the return value is referenced. In the shell, we still can access it using `v(N)` command. I tried `f()` command, and it only unbinds the variables. So what's the mechanism of Erlang shell gc? Thanks. BR On Sun, Jan 15, 2017 at 2:09 PM, Jack Tang wrote: > Hi Daniel > > Thank you for the clue and excellent recon toolkits! I would like dig it > deeper :) > > BR > -Jack > > On Sun, Jan 15, 2017 at 4:47 AM, D?niel Szoboszlay > wrote: > >> Hi Jack, >> >> I guess the 9 GB is lost due to memory fragmentation. Erlang allocates >> memory in large chunks called carriers from the OS, then places the blocks >> your program actually needs on these carriers. A carrier can only be >> returned to the OS once all the blocks on it have been freed (and even >> then, the memory allocator may decide to keep it around for a while in case >> more memory is needed). >> >> You can check with recon_alloc >> how much unused memory >> is lost due to fragmentation in the various allocators. >> >> The bad news is that you cannot defragment the carriers, and if the >> selected memory allocator strategy doesn't work well for your application, >> you cannot change it either without restarting the emulator. >> >> However, if the memory is wasted in the eheap_alloc, you may try to >> force a GC on all processes a couple of times. As the GC copies the memory, >> it will allocate new blocks and free up the old heap blocks. So there's a >> chance the allocators can compact the blocks together on fewer segments. >> But that's just a guess, it may or may not work at all. >> >> Cheers, >> Daniel >> >> On Sat, 14 Jan 2017 at 08:04 Jack Tang wrote: >> >>> Hello list, >>> >>> I run one Erlang application on Debian server and today I find the beam >>> process consumes around 35G memory by `top` command. >>> >>> ``` >>> KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers >>> KiB Swap: 0 total, 0 used, 0 free. 18684864 cached >>> Mem >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>> COMMAND >>> 11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp >>> ``` >>> >>> I connect to the Erlang application using remote shell and find the >>> mem-leaked supervisor tree and run gc on the whole tree. Code looks like >>> blow: >>> >>> ``` >>> lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) >>> end, supervisor:which_children(some_thing_sup)). >>> ``` >>> >>> and erlang:memory() decreases from 32G to 23G. >>> ``` >>> [{total,22982011544}, >>> {processes,12884182336}, >>> {processes_used,12884170336}, >>> {system,10097829208}, >>> {atom,13828705}, >>> {atom_used,13796692}, >>> {binary,170530288}, >>> {code,16450626}, >>> {ets,9637717576}] >>> ``` >>> >>> However, when I input `top` command, the beam process still takes 35G >>> memory. What can I do to release the 9G memory? Thanks >>> >>> BR >>> -Jack >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > > > -- > Jack Tang > > > http://www.linkedin.com/in/jacktang > -- Jack Tang http://www.linkedin.com/in/jacktang -------------- next part -------------- An HTML attachment was scrubbed... URL: From dszoboszlay@REDACTED Mon Jan 23 11:30:54 2017 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Mon, 23 Jan 2017 10:30:54 +0000 Subject: [erlang-questions] Overloaded function specifications as types Message-ID: A function's type specification can be overloaded: -spec foo(integer()) -> integer() ; (atom()) -> atom(). Is there a way I can use this overloaded function's type in non-spec contexts? Like I want to say bar/1 takes a function as an argument that has a spec like foo/1 above. I can think of the following options, but neither of them is exactly what I want: -spec bar(fun((integer() | atom()) -> integer() | atom())) -> ok. %% Allows a function taking an integer and returning an atom. -spec bar(fun((integer()) -> integer()) | fun((atom()) -> atom())) -> ok. %% Allows fun erlang:atom_to_list/1, which doesn't work on integers at all. Any better ideas? I'm sure Dialyzer can somehow represent internally the overloaded function types, but is there a way I can write them myself? Tanks, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Mon Jan 23 11:38:29 2017 From: alex0player@REDACTED (Alex S.) Date: Mon, 23 Jan 2017 13:38:29 +0300 Subject: [erlang-questions] Overloaded function specifications as types In-Reply-To: References: Message-ID: > 23 ???. 2017 ?., ? 13:30, D?niel Szoboszlay ???????(?): > > A function's type specification can be overloaded: > > -spec foo(integer()) -> integer() > ; (atom()) -> atom(). > > Is there a way I can use this overloaded function's type in non-spec contexts? Like I want to say bar/1 takes a function as an argument that has a spec like foo/1 above. > > I can think of the following options, but neither of them is exactly what I want: > > -spec bar(fun((integer() | atom()) -> integer() | atom())) -> ok. > %% Allows a function taking an integer and returning an atom. > > -spec bar(fun((integer()) -> integer()) | fun((atom()) -> atom())) -> ok. > %% Allows fun erlang:atom_to_list/1, which doesn't work on integers at all. Unfortunately, that requires having contravariant first-class types, which Dialyzer doesn?t support by design currently. Best thing you can do in your case is writing fun((X) -> X) when X :: atom(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Mon Jan 23 11:58:47 2017 From: kostis@REDACTED (Kostis Sagonas) Date: Mon, 23 Jan 2017 11:58:47 +0100 Subject: [erlang-questions] Overloaded function specifications as types In-Reply-To: References: Message-ID: <9884bbd0-f1f3-893f-1435-bdeabc6dc334@cs.ntua.gr> On 01/23/2017 11:30 AM, D?niel Szoboszlay wrote: > Is there a way I can use this overloaded function's type in non-spec > contexts? Like I want to say bar/1 takes a function as an argument that > has a spec like foo/1 above. > > I can think of the following options, but neither of them is exactly > what I want: > >... > > -spec bar(fun((integer()) -> integer()) | fun((atom()) -> atom())) -> ok. > %% Allows fun erlang:atom_to_list/1, which doesn't work on integers at all. I'm seriously confused as I do not see anything related to lists here... Can you elaborate? Kostis From dszoboszlay@REDACTED Mon Jan 23 12:32:05 2017 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Mon, 23 Jan 2017 11:32:05 +0000 Subject: [erlang-questions] Overloaded function specifications as types In-Reply-To: <9884bbd0-f1f3-893f-1435-bdeabc6dc334@cs.ntua.gr> References: <9884bbd0-f1f3-893f-1435-bdeabc6dc334@cs.ntua.gr> Message-ID: > > > -spec bar(fun((integer()) -> integer()) | fun((atom()) -> atom())) -> ok. > > %% Allows fun erlang:atom_to_list/1, which doesn't work on integers at > all. > > I'm seriously confused as I do not see anything related to lists here... > > Can you elaborate? > > I'm sorry, editing mistake! atom_to_list is of course not a good example. (I started off with different code samples and edited them a couple of times.) The correct example would be a function that only takes atoms and returns atoms. E.g. good_example(ok) -> ok; good_example(Other) when is_atom(Other) -> error. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Mon Jan 23 13:28:49 2017 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 23 Jan 2017 13:28:49 +0100 Subject: [erlang-questions] Erlang 19 callbacks aren't attributes? In-Reply-To: References: Message-ID: On Mon, Jan 16, 2017 at 8:08 PM, Ryan Stewart wrote: > There was a question about this on the list a few months back, but it was > unanswered. It seems that in Erlang 19, a module's callbacks no longer > appear in the module_info() as 'attributes'. They are, however, included in > the abstract code (via beam_lib). I'm wondering if this was an intentional > change or an oversight. The change was intentional, but we did not foresee that it would cause problems. There is no formal documentation exactly which attributes should appear in module_info(attributes). A long time ago, all attributes except 'module', 'export', 'import', 'file', and 'record' were included. When the dialyzer attributes 'type', 'spec', and 'opaque' were introduced we decided not to include them to reduce the code size. Since 'callback' and 'optional_callback' are attributes that are used by dialyzer, we decided that they too should be excluded (for consistency). I don't know exactly what information meck needs, but there is the behaviour_info/1 function can be used to retrieve the names of callback functions. See: http://erlang.org/doc/design_principles/spec_proc.html#id81417 If the meck will need the complete spec, the solution for now at least is to extract from the abstract code. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From garryh@REDACTED Mon Jan 23 18:10:18 2017 From: garryh@REDACTED (Garry Hodgson) Date: Mon, 23 Jan 2017 12:10:18 -0500 Subject: [erlang-questions] managing reconnects in alternate http clients Message-ID: <4d813c03-62bb-fe5f-eb59-da810791d49a@att.com> We've been looking at replacing our use of httpc with some other http client like Hackney or Gun. Both of those clients have the feature that they keep live connections to servers, reconnecting as needed. This is generally a Good Thing, but is problematic in our environment. We are using SDP (Software Defined Perimeter), which requires that we send a special SPA packet to effectively do a port knock to open port before connecting. The port knock opens a pinhole firewall on server side that will allow connect from client for a short time. Otherwise all connects are ignored (drop all firewall rule). So my question is, can this reconnect feature be disabled in either Gun or Hackney? Or is there some hook we could use to pass a "grease the skids" fun to be called first when reconnecting? From zzantozz@REDACTED Tue Jan 24 04:04:48 2017 From: zzantozz@REDACTED (Ryan Stewart) Date: Tue, 24 Jan 2017 03:04:48 +0000 Subject: [erlang-questions] Erlang 19 callbacks aren't attributes? In-Reply-To: References: Message-ID: On Mon, Jan 23, 2017 at 6:28 AM Bj?rn Gustavsson wrote: > The change was intentional, but we did not foresee that it would > cause problems. > > There is no formal documentation exactly which attributes should > appear in module_info(attributes). A long time ago, all attributes > except 'module', 'export', 'import', 'file', and 'record' were included. > > Well, to that point, the following appears on the Modules reference page ( http://erlang.org/doc/reference_manual/modules.html): "Any module attribute can be specified. The attributes are stored in the compiled code and can be retrieved by calling Module:module_info(attributes), or by using the module beam_lib(3) in STDLIB." The most straightforward understanding of that has to be that all attributes are equally accessible in both places. When the dialyzer attributes 'type', 'spec', and 'opaque' were introduced > we decided not to include them to reduce the code size. > > Since 'callback' and 'optional_callback' are attributes that > are used by dialyzer, we decided that they too should be > excluded (for consistency). > > That makes sense in a way, but at the same time, it seems very odd for all of these things to be "attributes", and yet only some of them appear when you ask the language for the attributes of a module. Regardless of the intended use of an attribute, why would the platform lie about whether or not an attribute is present? Also, what do you mean by "excluded"? As I pointed out in my post, the attributes are still present in the abstract code. I don't understand what's saved by having them in one place but not the other. Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: From thom.cherryhomes@REDACTED Tue Jan 24 03:12:35 2017 From: thom.cherryhomes@REDACTED (Thom Cherryhomes) Date: Tue, 24 Jan 2017 02:12:35 +0000 Subject: [erlang-questions] Complex state machines for complex telnet applications In-Reply-To: References: Message-ID: Would you, for example, build the ranch protocol as simple as possible, and link the other application servers, as needed, with a common protocol? -Thom On Sun, Jan 22, 2017 at 2:56 PM Thom Cherryhomes wrote: > Hello, everyone. This is my first post to the list. > > > I?m a long time swiss army knife of a software and hardware engineer who > has started to pick up Erlang, because I believe it and languages like it > to be the future of software development. > > With that said, I am working on a personal project of mine that I have > wanted to complete, a 21st century version of a BBS system, and I believe > Erlang and OTP are a good language and toolkit to write it in. > > > Code is being dropped here: http://github.com/tschak909/nakbbs > > > I am experimenting with both Ranch and just using gen_tcp with my own > socket pools, but I do have some design questions. > > > The telnet ports need to present a definite series of states, broken up > into seperate pieces: > > > I have definite common events that can be stretched across multiple use > cases: > > > * Display > > * Wait for line of input > > * Wait for character input > > * branch to another menu > > * hang up > > > But there will also be definite sections of the system, which could be > thought of as states: > > > * Authentication > > * Message Browser > > * Message Editor > > * Various Games > > > etc. > > > and I?m trying to think of an appropriate architectural pattern that will > avoid me basically having to cram every conceivable state as a sort of > hierarchy of tuples into a single gen_statem module. How can I approach > this in a nice abstract and extensible manner, that I could for example > push into a Behaviour and subsequently utilize for different terminal > types? It would be nice to split up the different parts of the system into > applications that would essentially be called from well defined APIs? > > > I?m just trying to figure this all out, and would love a semblence of > direction so I can do the proper research and deeply understand what I need > to learn. > > > -Thom > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jan 24 12:47:52 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 24 Jan 2017 11:47:52 +0000 Subject: [erlang-questions] Complex state machines for complex telnet applications In-Reply-To: References: Message-ID: Here are some thoughts off-hand. They are not too fleshed out. * I'd definitely run things via Ranch if possible. Some of the subtler things in and around reconnect accept pools and so on happens to be solved well by Loic in Ranch. * It may be possible to generalize buffer handling from the user. Consider a function handle(Data, BufferState) -> {Cmds, UpdatedBufferState}, where Cmds is a list of the commands the user entered. * Most modules in a BBS probably runs in a Request -> Reply fashion. You send a command to the module and it replies with a new display state you send to the BBS client. Since each module will keep its own state, you can stuff that into a process. * Modules may have their own state. They may also have BBS-global state which enables communication between clients. Say you have a forum module. It will have a forum_backend in which you store the actual posts in the forum. When a client requests forum access, you spawn a helper which runs the Request/Reply pattern proxying the data in the backend to the client. * gen_statem is probably most useful in modules, and not so much in the client itself, unless the interactions are complex and there are certain things which can only happen in some limited states, etc. * Be aggressive and link everything or use monitors. That way, you can easily handle errors in parts of the system without taking it all down. Also, you can probably handle a hangup by simply terminating the client process and then have modules handle a 'DOWN' message and clean up. * Old BBS'es did not really have it that much, but you could write a general command parser and then supply parsed terms to modules. This becomes yet another state machine, but it often kept inside the client process: -record(state, { buffer_state :: buffer_state(), parser_state :: parser_state(), socket :: inet:socket(), ... }). * A common beginners mistake is to want to process everything en masse. The rule of thumb is to create a process for each truly concurrent activity in the system. With that in mind, it may be that module state should not by in a proxy process, as above, but rather be embedded directly in the client. The forum_backend is truly concurrent however and warrants its own process. * The counterpoint to the last remark: don't be afraid to use a process to isolate parts of your system and keep complexity down. Juggling many states in one process at the same time means you have to handle every complex cross-interaction between them. If you push something to its own process, then chances are there are less cross-talk and thus a system which is easier to understand and handle. * Finally: rather than focusing on the "behaviour", focus on the interaction protocol. When you talk HTTP, you don't do that by inheriting a HTTP-class. You do that by speaking the protocol. This is powerful because you *don't* know the internal implementation of the HTTP server. REST was only possible in the first place because HTTP was a protocol. In a BBS, focus on the module interaction protocol. You don't care for the internal implementation, but rather how you talk to it. Once established, it may be that 80% of the modules can implemented via a generic callback framework, as is the case for the gen_server (where the protocol is OTP). But there are always 20% which are not fitting in, and you want that to be possible as well. Feel free to follow up :) On Tue, Jan 24, 2017 at 6:02 AM Thom Cherryhomes wrote: > Would you, for example, build the ranch protocol as simple as possible, > and link the other application servers, as needed, with a common protocol? > > -Thom > > On Sun, Jan 22, 2017 at 2:56 PM Thom Cherryhomes < > thom.cherryhomes@REDACTED> wrote: > > Hello, everyone. This is my first post to the list. > > > I?m a long time swiss army knife of a software and hardware engineer who > has started to pick up Erlang, because I believe it and languages like it > to be the future of software development. > > With that said, I am working on a personal project of mine that I have > wanted to complete, a 21st century version of a BBS system, and I believe > Erlang and OTP are a good language and toolkit to write it in. > > > Code is being dropped here: http://github.com/tschak909/nakbbs > > > I am experimenting with both Ranch and just using gen_tcp with my own > socket pools, but I do have some design questions. > > > The telnet ports need to present a definite series of states, broken up > into seperate pieces: > > > I have definite common events that can be stretched across multiple use > cases: > > > * Display > > * Wait for line of input > > * Wait for character input > > * branch to another menu > > * hang up > > > But there will also be definite sections of the system, which could be > thought of as states: > > > * Authentication > > * Message Browser > > * Message Editor > > * Various Games > > > etc. > > > and I?m trying to think of an appropriate architectural pattern that will > avoid me basically having to cram every conceivable state as a sort of > hierarchy of tuples into a single gen_statem module. How can I approach > this in a nice abstract and extensible manner, that I could for example > push into a Behaviour and subsequently utilize for different terminal > types? It would be nice to split up the different parts of the system into > applications that would essentially be called from well defined APIs? > > > I?m just trying to figure this all out, and would love a semblence of > direction so I can do the proper research and deeply understand what I need > to learn. > > > -Thom > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Tue Jan 24 12:57:28 2017 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Tue, 24 Jan 2017 12:57:28 +0100 Subject: [erlang-questions] Erlang 19 callbacks aren't attributes? In-Reply-To: References: Message-ID: On Tue, Jan 24, 2017 at 4:04 AM, Ryan Stewart wrote: > Also, what do you mean by "excluded"? As I pointed out in my post, the > attributes are still present in the abstract code. I don't understand what's > saved by having them in one place but not the other. > Everything returned by module_info() is loaded into memory along with the code for the module. The abstract code will not loaded be loaded into memory. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From bchesneau@REDACTED Tue Jan 24 13:47:22 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 24 Jan 2017 12:47:22 +0000 Subject: [erlang-questions] managing reconnects in alternate http clients In-Reply-To: <4d813c03-62bb-fe5f-eb59-da810791d49a@att.com> References: <4d813c03-62bb-fe5f-eb59-da810791d49a@att.com> Message-ID: Garry this probably could be done in hackney bay connecting yourself to the server and pass your own socket. Can you describe more in an hackney ticket (or here) what is the workflow? - benoit On Mon, Jan 23, 2017 at 6:10 PM Garry Hodgson wrote: > We've been looking at replacing our use of httpc with some other http > client like Hackney or Gun. Both of those clients have the feature that > they keep live connections to servers, reconnecting as needed. This is > generally a Good Thing, but is problematic in our environment. We are > using SDP (Software Defined Perimeter), which requires that we send a > special SPA packet to effectively do a port knock to open port before > connecting. The port knock opens a pinhole firewall on server side that > will allow connect from client for a short time. Otherwise all connects > are ignored (drop all firewall rule). > > So my question is, can this reconnect feature be disabled in either Gun > or Hackney? Or is there some hook we could use to pass a "grease the > skids" fun to be called first when reconnecting? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thom.cherryhomes@REDACTED Tue Jan 24 18:24:07 2017 From: thom.cherryhomes@REDACTED (Thom Cherryhomes) Date: Tue, 24 Jan 2017 17:24:07 +0000 Subject: [erlang-questions] Complex state machines for complex telnet applications In-Reply-To: References: Message-ID: Thanks so much for replying! I've definitely got a ton of thinking ahead, but your comments have verified what little I have been able to figure out, thus far. The irony here is that I think the telnet part of the system will be the most complex, as it has to convey a highly stateful and interactive graph... a web frontend, or a json API for a thick client will be infinitely easier to deal with when the time comes...heheh... I'll try to do a pattern where each part of the BBS is its own application, with a well defined protocol, that a proxy links to and requests a new process for... ...as well as remove gen_statem from the client, and turn it into a gen_server, leaving gen_statem for the "other" side... To tell a bit more about what I'm writing: This is a sort of 21st century equivalent to a BBS system, think a BBS that has learned the lessons from today's social networks, and provides multiple front-ends for computers vintage and new alike, where: * frontends for telnet * ASCII, ANSI, PETSCII, ATASCII, etc..etc. * web frontend * JSON API for thick clients * Users create their own boards, public, or private * Ability to utilize distributed model to replicate boards to other BBSes * Chat * Games Yeah, it may seem quaint, but i've wanted to write a board of my own for about 25 years, but never found the time. The combination of learning Erlang, and seeing what it could provide to facilitate this idea, is what has spurred me into finally implementing it. I do not intend to make any money off of it, and will release it GPL, but I want to make an actual, real example of a _LARGE_ Erlang system, so people can see how one would be put together. -Thom On Tue, Jan 24, 2017 at 5:48 AM Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > Here are some thoughts off-hand. They are not too fleshed out. > > * I'd definitely run things via Ranch if possible. Some of the subtler > things in and around reconnect accept pools and so on happens to be solved > well by Loic in Ranch. > > * It may be possible to generalize buffer handling from the user. Consider > a function handle(Data, BufferState) -> {Cmds, UpdatedBufferState}, where > Cmds is a list of the commands the user entered. > > * Most modules in a BBS probably runs in a Request -> Reply fashion. You > send a command to the module and it replies with a new display state you > send to the BBS client. Since each module will keep its own state, you can > stuff that into a process. > > * Modules may have their own state. They may also have BBS-global state > which enables communication between clients. Say you have a forum module. > It will have a forum_backend in which you store the actual posts in the > forum. When a client requests forum access, you spawn a helper which runs > the Request/Reply pattern proxying the data in the backend to the client. > > * gen_statem is probably most useful in modules, and not so much in the > client itself, unless the interactions are complex and there are certain > things which can only happen in some limited states, etc. > > * Be aggressive and link everything or use monitors. That way, you can > easily handle errors in parts of the system without taking it all down. > Also, you can probably handle a hangup by simply terminating the client > process and then have modules handle a 'DOWN' message and clean up. > > * Old BBS'es did not really have it that much, but you could write a > general command parser and then supply parsed terms to modules. This > becomes yet another state machine, but it often kept inside the client > process: > > -record(state, { > buffer_state :: buffer_state(), > parser_state :: parser_state(), > socket :: inet:socket(), > ... > }). > > * A common beginners mistake is to want to process everything en masse. > The rule of thumb is to create a process for each truly concurrent activity > in the system. With that in mind, it may be that module state should not by > in a proxy process, as above, but rather be embedded directly in the > client. The forum_backend is truly concurrent however and warrants its own > process. > > * The counterpoint to the last remark: don't be afraid to use a process to > isolate parts of your system and keep complexity down. Juggling many states > in one process at the same time means you have to handle every complex > cross-interaction between them. If you push something to its own process, > then chances are there are less cross-talk and thus a system which is > easier to understand and handle. > > * Finally: rather than focusing on the "behaviour", focus on the > interaction protocol. When you talk HTTP, you don't do that by inheriting a > HTTP-class. You do that by speaking the protocol. This is powerful because > you *don't* know the internal implementation of the HTTP server. REST was > only possible in the first place because HTTP was a protocol. In a BBS, > focus on the module interaction protocol. You don't care for the internal > implementation, but rather how you talk to it. Once established, it may be > that 80% of the modules can implemented via a generic callback framework, > as is the case for the gen_server (where the protocol is OTP). But there > are always 20% which are not fitting in, and you want that to be possible > as well. > > Feel free to follow up :) > > > On Tue, Jan 24, 2017 at 6:02 AM Thom Cherryhomes < > thom.cherryhomes@REDACTED> wrote: > > Would you, for example, build the ranch protocol as simple as possible, > and link the other application servers, as needed, with a common protocol? > > -Thom > > On Sun, Jan 22, 2017 at 2:56 PM Thom Cherryhomes < > thom.cherryhomes@REDACTED> wrote: > > Hello, everyone. This is my first post to the list. > > > I?m a long time swiss army knife of a software and hardware engineer who > has started to pick up Erlang, because I believe it and languages like it > to be the future of software development. > > With that said, I am working on a personal project of mine that I have > wanted to complete, a 21st century version of a BBS system, and I believe > Erlang and OTP are a good language and toolkit to write it in. > > > Code is being dropped here: http://github.com/tschak909/nakbbs > > > I am experimenting with both Ranch and just using gen_tcp with my own > socket pools, but I do have some design questions. > > > The telnet ports need to present a definite series of states, broken up > into seperate pieces: > > > I have definite common events that can be stretched across multiple use > cases: > > > * Display > > * Wait for line of input > > * Wait for character input > > * branch to another menu > > * hang up > > > But there will also be definite sections of the system, which could be > thought of as states: > > > * Authentication > > * Message Browser > > * Message Editor > > * Various Games > > > etc. > > > and I?m trying to think of an appropriate architectural pattern that will > avoid me basically having to cram every conceivable state as a sort of > hierarchy of tuples into a single gen_statem module. How can I approach > this in a nice abstract and extensible manner, that I could for example > push into a Behaviour and subsequently utilize for different terminal > types? It would be nice to split up the different parts of the system into > applications that would essentially be called from well defined APIs? > > > I?m just trying to figure this all out, and would love a semblence of > direction so I can do the proper research and deeply understand what I need > to learn. > > > -Thom > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raghav.karol@REDACTED Tue Jan 24 23:02:12 2017 From: raghav.karol@REDACTED (Raghav Karol) Date: Tue, 24 Jan 2017 23:02:12 +0100 Subject: [erlang-questions] peculiar type spec Message-ID: I recently came across this type specification -type index_spec() :: {add, Index, SecondaryKey} | {remove, Index, SecondaryKey}. * Is this a `type variable', see [1] and how does one interpret it? Best, Raghav [1] http://erlang.org/doc/reference_manual/typespec.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From dima_kakurin@REDACTED Wed Jan 25 00:38:02 2017 From: dima_kakurin@REDACTED (Dmitry Kakurin) Date: Tue, 24 Jan 2017 23:38:02 +0000 Subject: [erlang-questions] Message send guarantees Message-ID: When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node? Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg? The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node: 1. A --msg1--> C (process A sends msg1 to process C) 2. A --msg2--> B 3. B --msg2--> C (process B simply proxies msg2 from A to C) I need a guarantee that msg2 will always appear in C's queue after msg1. P.S. I've read the FAQ 10.8 and 10.9 :-) Thank you, Dmitry. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunp@REDACTED Wed Jan 25 09:30:47 2017 From: arunp@REDACTED (Arun) Date: Wed, 25 Jan 2017 14:00:47 +0530 Subject: [erlang-questions] Ntp support in erlang Message-ID: <58886237.2030504@utl.in> Hi, I am developing an erlang application which runs on a hardware device and the device is connected in a network where a dedicated ntp server is available. Now the difficulty what I am facing is that my system time is not getting synced with ntp server, is there any support available in erlang so that I can directly sync my system time with ntp server by directly providing ntp server address. Thanks in advance Arun From lukas@REDACTED Wed Jan 25 09:37:27 2017 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 25 Jan 2017 09:37:27 +0100 Subject: [erlang-questions] Message send guarantees In-Reply-To: References: Message-ID: Hello, On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin wrote: > When I execute "pid ! msg" and it returns, what are the guarantees if pid > and self are on the same node? > Can I assume that by the time it returns, msg is in the pid's queue? And > as a result all messages sent to pid afterwards will be queued after msg? > No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow. > The reason I'm asking is because I need a guarantee of proper message > ordering in the following scenario when all processes involved are local to > a node: > 1. A --msg1--> C (process A sends msg1 to process C) > 2. A --msg2--> B > 3. B --msg2--> C (process B simply proxies msg2 from A to C) > > I need a guarantee that msg2 will always appear in C's queue after msg1. > You have to do this using messages or some other sync mechanism. > > P.S. I've read the FAQ 10.8 and 10.9 :-) > Please add to the FAQ if you think that you can make it clearer on this point. https://github.com/matthiasl/Erlang-FAQ Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Wed Jan 25 09:57:11 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 25 Jan 2017 09:57:11 +0100 Subject: [erlang-questions] peculiar type spec In-Reply-To: References: Message-ID: If you refer to that whole line you quoted, then no, that's a "type declaration". "Index" and "SecondaryKey" are just names that act as placeholders, and since there is no other type information along with them, those fields could have any type. A "type variable" is a variable used as a parameter of the declared type, as in this example (see section 7.3 in the link you included): -type orddict(Key, Val) :: [{Key, Val}]. Here, Key and Val are both type variables. You could for exampe use this to define another type, like so: -type my_dict() :: orddict(atom(), integer()). /Richard 2017-01-24 23:02 GMT+01:00 Raghav Karol : > I recently came across this type specification > > -type index_spec() :: {add, Index, SecondaryKey} | {remove, Index, > SecondaryKey}. > > * Is this a `type variable', see [1] and how does one interpret it? > > Best, > Raghav > > [1] http://erlang.org/doc/reference_manual/typespec.html > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Wed Jan 25 11:00:54 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 25 Jan 2017 11:00:54 +0100 Subject: [erlang-questions] Erlang 19 callbacks aren't attributes? In-Reply-To: References: Message-ID: There are some distinctions that have not been well documented. I'll try to clarify them based on how things actually work. First, an Erlang program is divided into "forms" - sequences of tokens that are terminated by a full stop. Forms are either function definitions (starting with a function name, i.e., an atom) or what's generally called "attributes", which start with a minus and an atom, as in '-module(..).'. Seems straightforward, right? There are however three kinds of forms in Erlang that have the general shape of attributes: * Preprocessor Directives - things like '-include("foo.hrl").' or '-ifdef(DEBUG)'. These only make sense to the preprocessor (epp), and are removed by the preprocessing step. * Declarations - things like module name, export/import lists, record definitions, type specifications, type definitions, and behaviour callbacks. Also some more unknown special declarations like '-file(...).' that the preprocessor inserts to tell the compiler which parts of the code came from which files, or errors and warnings passed on from preprocessing. The common thing about Declarations is that they have meaning on the Erlang Language level, and typically have particular rules for how they may look (a module declaration must provide a single module name, an import declaration must provide a module name and a list of function/arity pairs, a record declaration must list fields and possibly types, and so on), and where in the file they may occur (before any function, or interspersed with functions). Since these are part of the language, they are not included in the 'attributes' list of the Beam file; the corresponding information is generally found in some other part of the file. * "Wild" Attributes (as they were named in the draft Standard Erlang specification) - any attribute that is not one of the Declarations, and has the form '-Atom(Term)' - note that only one argument is allowed. These can occur anywhere in the file and have no special meaning except possibly to various tools. For example, '-compile(Options).' only has a meaning to the specific Erlang compiler that you're using; it's not part of the language. These are just annotations, and these are the ones that are listed in the 'attributes' section of the module info. These distinctions have not been properly documented, but have been intuitively followed by the people working on the language and compiler. In some cases, the lines are blurred. For example, one could easily argue that '-behaviour(Name).' was just another wild attribute which only had a meaning to the current version of the compiler, and not part of the language proper. On the other hand, since the -callback declarations were introduced, it's pretty clear that behaviours are a language feature. Along these lines, I think that e.g. the type declarations of a module should be included in Beam files as standard chunks, so you don't need to extract them from the "abstract format" debug information, if present, but that's another can of pull requests. /Richard 2017-01-24 4:04 GMT+01:00 Ryan Stewart : > On Mon, Jan 23, 2017 at 6:28 AM Bj?rn Gustavsson wrote: > >> The change was intentional, but we did not foresee that it would >> cause problems. >> >> There is no formal documentation exactly which attributes should >> appear in module_info(attributes). A long time ago, all attributes >> except 'module', 'export', 'import', 'file', and 'record' were included. >> >> Well, to that point, the following appears on the Modules reference page ( > http://erlang.org/doc/reference_manual/modules.html): > > "Any module attribute can be specified. The attributes are stored in the > compiled code and can be retrieved by calling Module:module_info(attributes), > or by using the module beam_lib(3) in STDLIB." > > The most straightforward understanding of that has to be that all > attributes are equally accessible in both places. > > When the dialyzer attributes 'type', 'spec', and 'opaque' were introduced >> we decided not to include them to reduce the code size. >> >> Since 'callback' and 'optional_callback' are attributes that >> are used by dialyzer, we decided that they too should be >> excluded (for consistency). >> >> That makes sense in a way, but at the same time, it seems very odd for > all of these things to be "attributes", and yet only some of them appear > when you ask the language for the attributes of a module. Regardless of the > intended use of an attribute, why would the platform lie about whether or > not an attribute is present? > > Also, what do you mean by "excluded"? As I pointed out in my post, the > attributes are still present in the abstract code. I don't understand > what's saved by having them in one place but not the other. > > Ryan > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Wed Jan 25 11:11:18 2017 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Wed, 25 Jan 2017 11:11:18 +0100 Subject: [erlang-questions] Complex state machines for complex telnet applications In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: From alex0player@REDACTED Wed Jan 25 11:22:31 2017 From: alex0player@REDACTED (Alex S.) Date: Wed, 25 Jan 2017 13:22:31 +0300 Subject: [erlang-questions] Erlang 19 callbacks aren't attributes? In-Reply-To: References: Message-ID: > 25 ???. 2017 ?., ? 13:00, Richard Carlsson ???????(?): > > There are some distinctions that have not been well documented. I'll try to clarify them based on how things actually work. > > First, an Erlang program is divided into "forms" - sequences of tokens that are terminated by a full stop. Forms are either function definitions (starting with a function name, i.e., an atom) or what's generally called "attributes", which start with a minus and an atom, as in '-module(..).'. Seems straightforward, right? > > There are however three kinds of forms in Erlang that have the general shape of attributes: > > * Preprocessor Directives - things like '-include("foo.hrl").' or '-ifdef(DEBUG)'. These only make sense to the preprocessor (epp), and are removed by the preprocessing step. > > * Declarations - things like module name, export/import lists, record definitions, type specifications, type definitions, and behaviour callbacks. Also some more unknown special declarations like '-file(...).' that the preprocessor inserts to tell the compiler which parts of the code came from which files, or errors and warnings passed on from preprocessing. The common thing about Declarations is that they have meaning on the Erlang Language level, and typically have particular rules for how they may look (a module declaration must provide a single module name, an import declaration must provide a module name and a list of function/arity pairs, a record declaration must list fields and possibly types, and so on), and where in the file they may occur (before any function, or interspersed with functions). Since these are part of the language, they are not included in the 'attributes' list of the Beam file; the corresponding information is generally found in some other part of the file. > > * "Wild" Attributes (as they were named in the draft Standard Erlang specification) - any attribute that is not one of the Declarations, and has the form '-Atom(Term)' - note that only one argument is allowed. These can occur anywhere in the file and have no special meaning except possibly to various tools. For example, '-compile(Options).' only has a meaning to the specific Erlang compiler that you're using; it's not part of the language. These are just annotations, and these are the ones that are listed in the 'attributes' section of the module info. > > These distinctions have not been properly documented, but have been intuitively followed by the people working on the language and compiler. In some cases, the lines are blurred. For example, one could easily argue that '-behaviour(Name).' was just another wild attribute which only had a meaning to the current version of the compiler, and not part of the language proper. On the other hand, since the -callback declarations were introduced, it's pretty clear that behaviours are a language feature. > > Along these lines, I think that e.g. the type declarations of a module should be included in Beam files as standard chunks, so you don't need to extract them from the "abstract format" debug information, if present, but that's another can of pull requests. Along those exact lines, I argue that -callback attributes are nothing more than a preprocessor directive that gets transformed into behaviour_info/1 function, and therefore has no place in the standard chunks. From kostis@REDACTED Wed Jan 25 11:57:12 2017 From: kostis@REDACTED (Kostis Sagonas) Date: Wed, 25 Jan 2017 11:57:12 +0100 Subject: [erlang-questions] Erlang 19 callbacks aren't attributes? In-Reply-To: References: Message-ID: <4e5dbb3e-2382-1d73-5fc0-43e43d934ec8@cs.ntua.gr> On 01/25/2017 11:22 AM, Alex S. wrote: >> 25 ???. 2017 ?., ? 13:00, Richard Carlsson ???????(?): >> >> There are some distinctions that have not been well documented. I'll try to clarify them based on how things actually work. ... ... >> These distinctions have not been properly documented, but have been intuitively followed by the people working on the language and compiler. In some cases, the lines are blurred. For example, one could easily argue that '-behaviour(Name).' was just another wild attribute which only had a meaning to the current version of the compiler, and not part of the language proper. On the other hand, since the -callback declarations were introduced, it's pretty clear that behaviours are a language feature. >> >> Along these lines, I think that e.g. the type declarations of a module should be included in Beam files as standard chunks, so you don't need to extract them from the "abstract format" debug information, if present, but that's another can of pull requests. > Along those exact lines, I argue that -callback attributes are nothing more than a preprocessor directive that gets transformed into behaviour_info/1 function, and therefore has no place in the standard chunks. You are simply wrong. For starters, -callback attributes are NOT related to the Erlang preprocessor (epp); the mechanism that creates the behaviour_info/1 function is in sys_pre_expand, which is part of the compiler. More importantly, callbacks are manipulated by other tools (e.g. dialyzer, also meck?), so clearly they _are_ "more than a preprocessor directive", even if this manipulation happens via the abstract code currently. But I think that the main problem with your mail (and my reply above!) is that it might unfocus the discussion about what is an attribute and what isn't. Richard Carlsson has written a wonderful mail which explains why certain things are the way they are, but also brings up two important (IMO) points that should not be lost: 1. "These distinctions have not been properly documented..." I think they should be. 2. Certain declarations need not be extracted from the "abstract format" debug information but should be included in Beam files as standard chunks. FWIW, I agree. Kostis From jesper.louis.andersen@REDACTED Wed Jan 25 13:09:21 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 25 Jan 2017 12:09:21 +0000 Subject: [erlang-questions] Ntp support in erlang In-Reply-To: <58886237.2030504@utl.in> References: <58886237.2030504@utl.in> Message-ID: I think you would normally run a separate NTP client which calls into the kernel and sets the time accordingly. There are some very small NTP clients out there. I know Poul Henning Kamp is currently planning to do some work in the area[0] and ntimed is already there and a fairly small NTP client (and only a client). As for the Erlang support, there is a couple of write-ups on how it handles the situation where the NTP client decides to move time suddenly. This is called a "warp", and how you configure your ERTS determines how it reacts. You have to pick the right solution for your situation: http://erlang.org/doc/apps/erts/time_correction.html http://learnyousomeerlang.com/time#time-warp [0] http://nwtime.org/projects/ntimed/ On Wed, Jan 25, 2017 at 9:31 AM Arun wrote: > Hi, > > I am developing an erlang application which runs on a hardware > device and the device is connected in a network where a dedicated ntp > server is available. Now the difficulty what I am facing is that my > system time is not getting synced with ntp server, is there any support > available in erlang so that I can directly sync my system time with ntp > server by directly providing ntp server address. > > Thanks in advance > Arun > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Wed Jan 25 11:24:22 2017 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Wed, 25 Jan 2017 11:24:22 +0100 Subject: [erlang-questions] Complex state machines for complex telnet applications In-Reply-To: Message-ID: <2fce03cf-7979-45f8-8c22-e0676954a13e@email.android.com> An HTML attachment was scrubbed... URL: From serge@REDACTED Wed Jan 25 15:33:42 2017 From: serge@REDACTED (Serge Aleynikov) Date: Wed, 25 Jan 2017 09:33:42 -0500 Subject: [erlang-questions] Ntp support in erlang In-Reply-To: <58886237.2030504@utl.in> References: <58886237.2030504@utl.in> Message-ID: A while back I had to deal with the same issue and put together this module that obtained NTP time from servers: https://github.com/saleyn/util/blob/master/src/sntp.erl Regards, Serge On Wed, Jan 25, 2017 at 3:30 AM, Arun wrote: > Hi, > > I am developing an erlang application which runs on a hardware device > and the device is connected in a network where a dedicated ntp server is > available. Now the difficulty what I am facing is that my system time is > not getting synced with ntp server, is there any support available in > erlang so that I can directly sync my system time with ntp server by > directly providing ntp server address. > > Thanks in advance > Arun > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Wed Jan 25 17:14:12 2017 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Wed, 25 Jan 2017 16:14:12 +0000 Subject: [erlang-questions] enif_send rules questions In-Reply-To: References: <612e8bad-f9cb-312d-0c90-f03d9f1ebe2f@ericsson.com> Message-ID: I see another scenario here: What happens in the context of enif_send() in a scoped thread? That is, in my NIF function I spin up a thread that does an enif_send() and exits before my NIF function returns. The process environment is in fact in scope for the enif_send, but I'm also in a created thread. Should I plug in the process environment or null? On Sat, Jan 14, 2017 at 1:04 AM Jason Orendorff wrote: > On Fri, Jan 13, 2017 at 10:09 AM, Sverker Eriksson < > sverker.eriksson@REDACTED> wrote: > > On 01/12/2017 08:21 PM, Jason Orendorff wrote: > > 4. We suspect it's actually totally safe to pass a process-independent > environment as the first parameter, regardless of whether there's a > "calling process" or we're "calling from a created thread". Is it? > If so, would you accept a patch to document that that's OK? > > I'm not super excited about such a guarantee, at least not right now. > > > Thank you for the quick, thoughtful response. I appreciate your time. > > Is there a way to enforce this rule for our users that we've missed? When > our `send` method is called, I think we have to figure out if there's a > current > calling process, and if so, its environment. I don't see a way to do it > without > storing that information in TLS ahead of time -- at a cost to every NIF > call. > > We could make code that runs on created threads statically different from > code that runs in the Erlang scheduler's threads, e.g. by making > `NifCallEnv`, `AllocatedEnvInNifThread`, and `AllocatedEnvInCreatedThread` > three different types, of which only 2 have `.send()` methods. But this > seems horrible. > > We have considered several possible approaches, all about as bad as that. > > Thanks, > -j > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Wed Jan 25 18:24:16 2017 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Wed, 25 Jan 2017 18:24:16 +0100 Subject: [erlang-questions] enif_send rules questions In-Reply-To: References: <612e8bad-f9cb-312d-0c90-f03d9f1ebe2f@ericsson.com> Message-ID: <4884c863-9086-daeb-cc05-4ceb9fef8c25@ericsson.com> The lifetime of the thread doesn't matter. "env The environment of the calling process. Must be NULL only if calling from a created thread." enif_send(env, ...) uses argument 'env' to make assumptions about the thread context, such as seized locks for example. When calling from an Erlang process, the main lock of the process is held. /Sverker On 01/25/2017 05:14 PM, Daniel Goertzen wrote: > I see another scenario here: What happens in the context of enif_send() in > a scoped thread? That is, in my NIF function I spin up a thread that does > an enif_send() and exits before my NIF function returns. The process > environment is in fact in scope for the enif_send, but I'm also in a > created thread. Should I plug in the process environment or null? > > On Sat, Jan 14, 2017 at 1:04 AM Jason Orendorff > wrote: > >> On Fri, Jan 13, 2017 at 10:09 AM, Sverker Eriksson < >> sverker.eriksson@REDACTED> wrote: >> >> On 01/12/2017 08:21 PM, Jason Orendorff wrote: >> >> 4. We suspect it's actually totally safe to pass a process-independent >> environment as the first parameter, regardless of whether there's a >> "calling process" or we're "calling from a created thread". Is it? >> If so, would you accept a patch to document that that's OK? >> >> I'm not super excited about such a guarantee, at least not right now. >> >> >> Thank you for the quick, thoughtful response. I appreciate your time. >> >> Is there a way to enforce this rule for our users that we've missed? When >> our `send` method is called, I think we have to figure out if there's a >> current >> calling process, and if so, its environment. I don't see a way to do it >> without >> storing that information in TLS ahead of time -- at a cost to every NIF >> call. >> >> We could make code that runs on created threads statically different from >> code that runs in the Erlang scheduler's threads, e.g. by making >> `NifCallEnv`, `AllocatedEnvInNifThread`, and `AllocatedEnvInCreatedThread` >> three different types, of which only 2 have `.send()` methods. But this >> seems horrible. >> >> We have considered several possible approaches, all about as bad as that. >> >> Thanks, >> -j >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> From jason.orendorff@REDACTED Wed Jan 25 19:55:48 2017 From: jason.orendorff@REDACTED (Jason Orendorff) Date: Wed, 25 Jan 2017 12:55:48 -0600 Subject: [erlang-questions] enif_send rules questions In-Reply-To: <4884c863-9086-daeb-cc05-4ceb9fef8c25@ericsson.com> References: <612e8bad-f9cb-312d-0c90-f03d9f1ebe2f@ericsson.com> <4884c863-9086-daeb-cc05-4ceb9fef8c25@ericsson.com> Message-ID: Quick follow-up: I found out I could use enif_thread_type() to detect cases where my users would be breaking the documented contract, and error out instead. -j -------------- next part -------------- An HTML attachment was scrubbed... URL: From cean.ebengt@REDACTED Wed Jan 25 22:06:11 2017 From: cean.ebengt@REDACTED (bengt) Date: Wed, 25 Jan 2017 22:06:11 +0100 Subject: [erlang-questions] Looking for Erlang code that handles PDF file attachments Message-ID: Greetings, I would like to attach files (or some sort of binary content, anyway) to PDF files, without destroying the PDF'ness of the file. I would also like to be able to read the created PDF file and be able to tell what is the attachment (and what is the original content). I have found Itext (Java, see http://itextpdf.com/ ). It seems to be able to do that and everything else too, but is there any other alternative? Preferably in Erlang. bengt -------------- next part -------------- An HTML attachment was scrubbed... URL: From charles.shuller@REDACTED Wed Jan 25 23:15:30 2017 From: charles.shuller@REDACTED (Charles Shuller) Date: Wed, 25 Jan 2017 16:15:30 -0600 Subject: [erlang-questions] module vs gen_server Message-ID: What things should be considered when deciding if a chunk of functionality should be implemented as a plain old module or a gen_server?? Thanks! Charles -------------- next part -------------- An HTML attachment was scrubbed... URL: From dima_kakurin@REDACTED Thu Jan 26 00:06:05 2017 From: dima_kakurin@REDACTED (Dmitry Kakurin) Date: Wed, 25 Jan 2017 23:06:05 +0000 Subject: [erlang-questions] Message send guarantees In-Reply-To: References: , Message-ID: Thanks Lukas, please see a couple of questions/comments below. - Dmitry ________________________________ From: Lukas Larsson Sent: Wednesday, January 25, 2017 12:37 AM Hello, On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin > wrote: When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node? Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg? No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow. What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why. The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node: 1. A --msg1--> C (process A sends msg1 to process C) 2. A --msg2--> B 3. B --msg2--> C (process B simply proxies msg2 from A to C) I need a guarantee that msg2 will always appear in C's queue after msg1. You have to do this using messages or some other sync mechanism. You are right, in general it's easy to devise a simple scheme for that. Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me). With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want. Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem? P.S. I've read the FAQ 10.8 and 10.9 :-) Please add to the FAQ if you think that you can make it clearer on this point. https://github.com/matthiasl/Erlang-FAQ Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From vances@REDACTED Thu Jan 26 06:53:10 2017 From: vances@REDACTED (Vance Shipley) Date: Thu, 26 Jan 2017 11:23:10 +0530 Subject: [erlang-questions] module vs gen_server In-Reply-To: References: Message-ID: On Thu, Jan 26, 2017 at 3:45 AM, Charles Shuller wrote: > What things should be considered when deciding if a chunk of functionality > should be implemented as a plain old module or a gen_server?? Think of the process(es) which will execute the code providing the functionality. If existing processes will call functions in your module and execute that code inline than it is a library module and needs no behavior. If the functionality will be provided by executing the code in a new process than you should consider how you will handle system messages in that process. If you don't yet know what that means you should use one of the standard behaviors which will provide everything you need to behave correctly in OTP. If you don't like to play it safe you can handle system messages in a "plain old module" (http://erlang.org/doc/man/sys.html). The more common question is: should I use a gen_server or a gen_fsm (or gen_statem) behavior? If you start with a gen_server and later realize you need to handle an event differently based on something in a previous event than your server has states and you should port it to a gen_fsm (or gen_statem). -- -Vance From raimo+erlang-questions@REDACTED Thu Jan 26 08:14:21 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 26 Jan 2017 08:14:21 +0100 Subject: [erlang-questions] Message send guarantees In-Reply-To: References: Message-ID: <20170126071421.GA46222@erix.ericsson.se> On Wed, Jan 25, 2017 at 11:06:05PM +0000, Dmitry Kakurin wrote: > Thanks Lukas, please see a couple of questions/comments below. > > - Dmitry > > ________________________________ > From: Lukas Larsson > Sent: Wednesday, January 25, 2017 12:37 AM > > Hello, > > On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin > wrote: > When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node? > Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg? > > No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow. > > What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why. On a multi-threade system it is very hard to know in which order things happen, so there is no "reordering" since there is no temporal order to begin with. Using the non-SMP virtual machine and erlang:yield/0 could do the trick, but depending on the computer you might loose boatloads of performance. It is also _very_ bad programming practice to pretend that asynchronous message passing is not asynchronous. It is solving a problem the wrong way... > > > The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node: > 1. A --msg1--> C (process A sends msg1 to process C) > 2. A --msg2--> B > 3. B --msg2--> C (process B simply proxies msg2 from A to C) > > I need a guarantee that msg2 will always appear in C's queue after msg1. > > You have to do this using messages or some other sync mechanism. > > You are right, in general it's easy to devise a simple scheme for that. > Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me). > With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want. If B can be modified as needed; you can get an ack from B so both A and B know about the bypass e.g to delay bypass until no messages are in flight, provided that the messages from B to C are acked. Or, maybe you can only use messages where the order does not matter. But without modifying your requirements as you state them I see no solution. Why do you really need to bypass some messages and still keep message ordering - it is impossible! > > Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem? From a process on the same node as C it is possible to inspect C's queue through erlang:process_info/2, but then you have started to dig a deep hole for yourself. You will get whole queue of C and have to traverse all of it to see if the message you sent is there. But just to inspect the queue will hurt message passing performance due to the locking involved. We have had customers that have used something like that in an attemt to do overload protection, only resulting in bad high load behaviour... > > > P.S. I've read the FAQ 10.8 and 10.9 :-) > > Please add to the FAQ if you think that you can make it clearer on this point. https://github.com/matthiasl/Erlang-FAQ > > Lukas > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From alex0player@REDACTED Thu Jan 26 09:38:24 2017 From: alex0player@REDACTED (Alex S.) Date: Thu, 26 Jan 2017 11:38:24 +0300 Subject: [erlang-questions] Message send guarantees In-Reply-To: <20170126071421.GA46222@erix.ericsson.se> References: <20170126071421.GA46222@erix.ericsson.se> Message-ID: > 26 ???. 2017 ?., ? 10:14, Raimo Niskanen ???????(?): > > On Wed, Jan 25, 2017 at 11:06:05PM +0000, Dmitry Kakurin wrote: >> Thanks Lukas, please see a couple of questions/comments below. >> >> - Dmitry >> >> ________________________________ >> From: Lukas Larsson > >> Sent: Wednesday, January 25, 2017 12:37 AM >> >> Hello, >> >> On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin >> wrote: >> When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node? >> Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg? >> >> No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow. >> >> What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why. > > On a multi-threade system it is very hard to know in which order things > happen, so there is no "reordering" since there is no temporal order > to begin with. > > Using the non-SMP virtual machine and erlang:yield/0 could do the trick, > but depending on the computer you might loose boatloads of performance. > > It is also _very_ bad programming practice to pretend that asynchronous > message passing is not asynchronous. It is solving a problem the > wrong way? I?d like to clarify that the order of messages *sent from A to B* is kept intact indeed, and later messages will arrive later. > > >> >> >> The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node: >> 1. A --msg1--> C (process A sends msg1 to process C) >> 2. A --msg2--> B >> 3. B --msg2--> C (process B simply proxies msg2 from A to C) >> >> I need a guarantee that msg2 will always appear in C's queue after msg1. >> >> You have to do this using messages or some other sync mechanism. >> >> You are right, in general it's easy to devise a simple scheme for that. >> Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me). >> With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want. > > If B can be modified as needed; you can get an ack from B so both A and B > know about the bypass e.g to delay bypass until no messages are in flight, > provided that the messages from B to C are acked. > > Or, maybe you can only use messages where the order does not matter. > > But without modifying your requirements as you state them I see no solution. > > Why do you really need to bypass some messages and still keep message > ordering - it is impossible! I would recommend using synchronous calls, perhaps with a proxy ?queue? process or whatnot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandrashekhar.mullaparthi@REDACTED Thu Jan 26 10:28:19 2017 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Thu, 26 Jan 2017 09:28:19 +0000 Subject: [erlang-questions] Looking for Erlang code that handles PDF file attachments In-Reply-To: References: Message-ID: Have you looked into erlguten? https://github.com/richcarl/erlguten cheers, Chandru On 25 January 2017 at 21:06, bengt wrote: > Greetings, > > I would like to attach files (or some sort of binary content, anyway) to > PDF files, without destroying the PDF'ness of the file. I would also like > to be able to read the created PDF file and be able to tell what is the > attachment (and what is the original content). I have found Itext (Java, > see http://itextpdf.com/). It seems to be able to do that and everything > else too, but is there any other alternative? Preferably in Erlang. > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Thu Jan 26 12:43:50 2017 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 26 Jan 2017 12:43:50 +0100 Subject: [erlang-questions] Looking for Erlang code that handles PDF file attachments In-Reply-To: References: Message-ID: I did exactly this a while back. The enclosed PDF has an erlang program packed inside it. The erlang program can be used to pack and unpack things inside PDF. PDF is after all, just a container format (like a zip file) Hint take a look at the end of erlpdf.pdf in a text editor. Actually since I always want to read the documentation first - I thought the code should be *inside* the documentation and NOT the other way around. Cheers /Joe On Wed, Jan 25, 2017 at 10:06 PM, bengt wrote: > Greetings, > > I would like to attach files (or some sort of binary content, anyway) to PDF > files, without destroying the PDF'ness of the file. I would also like to be > able to read the created PDF file and be able to tell what is the attachment > (and what is the original content). I have found Itext (Java, see > http://itextpdf.com/). It seems to be able to do that and everything else > too, but is there any other alternative? Preferably in Erlang. > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- A non-text attachment was scrubbed... Name: erlpdf.pdf Type: application/pdf Size: 51518 bytes Desc: not available URL: From mononcqc@REDACTED Thu Jan 26 14:09:41 2017 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 26 Jan 2017 08:09:41 -0500 Subject: [erlang-questions] Message send guarantees In-Reply-To: References: Message-ID: <20170126130940.GA41132@ferdmbp.local> On 01/25, Dmitry Kakurin wrote: > >The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node: >1. A --msg1--> C (process A sends msg1 to process C) >2. A --msg2--> B >3. B --msg2--> C (process B simply proxies msg2 from A to C) > >I need a guarantee that msg2 will always appear in C's queue after msg1. > > [...] > >Unfortunately I have a limitation. In my case A talks to C thru proxy B >and would like to short-circuit the communication by getting rid of B >and start talking to C directly. Process A needs to do that while >messages thru B could be in flight. The limitation is that C is not >aware of this proxy business, and cannot be changed (it's a regular >GenServer that is not written by me). >With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want. > >Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem? > As others have mentioned, this is not really possible. You *could* observe the queue, but it is not practical. The interesting thing with Erlang there is that a lot of these design decisions work the same whether the processes are local or remote. The problem you see here of message ordering is very similar to the one you get of packet ordering when delivering data over a network. Generally, the way to solve the problem is to label every message you send with a sequence number that increments. On every message received, the final receiver can then look at the sequence numbers and see if any message is missing. It can then block, ask for a retransmission, or disregard duplicated messages. Because Erlang's guarantees for message passing only involve any given pair of processes (A -> B, or A -> C) but not more, we get a case where the following race conditions are very possible: A B C | --- 1 ---> | | | --- 2 ---------------> | | | --- 1 --> | The stop at B means that through scheduling, the message can be delayed for unkown periods of time and the delays can mess up the ordering. The only surefire way of getting the messages to C in order is to either a) send them directly from a single sender every time, or b) to let C know if the messages are not in order so it can do the reordering itself. Anything else will be very hacky (implementing locks in userland and synchronizing everything) and would usually be more of a code smell than anything. Note: The usage of sequential number is a more specific implementation of causal ordering, where a logical 'stamp' is used to denote a 'happens-before' relationship between events. Vector clocks, version vectors, and interval tree clocks are other data structures that let you get more general ordering guarantees, but that much power isn't necessarily required here. From jesper.louis.andersen@REDACTED Thu Jan 26 14:28:18 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 26 Jan 2017 13:28:18 +0000 Subject: [erlang-questions] module vs gen_server In-Reply-To: References: Message-ID: On Thu, Jan 26, 2017 at 5:20 AM Charles Shuller wrote: > What things should be considered when deciding if a chunk of functionality > should be implemented as a plain old module or a gen_server?? > As a main rule: Stateless functionality goes into library modules, stateful functionality goes into a gen_server process. A good example is a database pool. The protocol encoder/decoder for the database is a library and should live separately. A stateful connection to the DB uses the protocol module to provide its functionality. Finally, a separate system handles the pooling and its guarantees. By splitting the system up, you can reuse the DB connection code in another pool. Or you can move the connection handling into another process by calling directly into the library. Another main rule: Each concurrent activity which needs to keep state must have its own process. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Thu Jan 26 14:29:50 2017 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 26 Jan 2017 14:29:50 +0100 Subject: [erlang-questions] Message send guarantees In-Reply-To: References: Message-ID: On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin wrote: > When I execute "pid ! msg" and it returns, what are the guarantees if pid > and self are on the same node? > Can I assume that by the time it returns, msg is in the pid's queue? And as > a result all messages sent to pid afterwards will be queued after msg? > > The reason I'm asking is because I need a guarantee of proper message > ordering in the following scenario when all processes involved are local to > a node: > 1. A --msg1--> C (process A sends msg1 to process C) > 2. A --msg2--> B > 3. B --msg2--> C (process B simply proxies msg2 from A to C) > > I need a guarantee that msg2 will always appear in C's queue after msg1. This cannot be guaranteed In the absence of an error you can assume the following: That sequences of messages between pairs of processes are ordered (ie they arrive in the same order they were sent) You will never know if messages are processed by the destination process Note I said "in the absence of an error" - to take care of the error case we use links - links tell you if the linked processes have crashed, nothing else. Even if a message arrives at a process you will never know, you will also never know if the processing it triggered worked - if you want to know send a message back and wait for it. If you want to know if a process works send it a message asking it to do something, wait for a reply and check the reply is OK. All this will tell you is that the process WAS working they last time you asked it - and NOT that it IS working now. Also remember that messages take time to propagate - and processes take time to do things In your A -> B followed by B->C B might take a long time between relaying the messages (due to garbage collection, or something weird) The above behaviour does not break any laws of physics - ie messages take time, causality is obeyed, these is no spooky action at a distance etc. Cheers /Joe > > P.S. I've read the FAQ 10.8 and 10.9 :-) > > Thank you, Dmitry. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From jesper.louis.andersen@REDACTED Thu Jan 26 14:41:46 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 26 Jan 2017 13:41:46 +0000 Subject: [erlang-questions] Message send guarantees In-Reply-To: References: Message-ID: On Thu, Jan 26, 2017 at 5:21 AM Dmitry Kakurin wrote: > What aspect of implementation can cause such reordering? Could you give an > example how this could happen? Maybe I'll be able to artificially create > conditions where it would not happen. Or work around it in some other way. > Please see below why. > > > A straightforward implementation would lock the mailbox of the receiving process, put a message there, unlock the mailbox and proceed. In such a system, the ordering is guaranteed because you have a happens-before relationship on the message send. If the process is remote, TCP buffering can reorder message arrival, so in general it is not good to structure systems under the assumption that there is a happens-before relationship between messages like this. However, there is also the need to satisfy future possible implementations. I could delay sending messages for a while and then send them as a large batch later on. This is allowed in the semantics since the ! operator works asynchronously. I could implement it as: Allocate a new message. Try a CAS operation on the target mailbox. If the CAS operation fails, delay the attempt to later due to contention on the target mailbox. In this setting, there is no happens-before on '!', and thus messages may be reordered. If you start relying on internal implementation structure, chances are your program subtly breaks later on. A good example was when the OTP team split up the single timer wheel into multiple timer wheels. Before, there were an order in which timers triggered and people relied on this order. But with multiple timer wheels, reorderings are possible and their code broke. My advice is to code for the rules of physics (causality). It tends to produce the best results in the long run. -------------- next part -------------- An HTML attachment was scrubbed... URL: From per@REDACTED Thu Jan 26 17:39:42 2017 From: per@REDACTED (Per Hedeland) Date: Thu, 26 Jan 2017 17:39:42 +0100 Subject: [erlang-questions] code path cache vs atom table Message-ID: <588A264E.4060607@hedeland.org> Hi, I guess this is more or less moot, since the code path cache has been removed in 19, but maybe this anecdote can be useful info for someone else that is still using it in older versions. We had a customer report that the emulator crashed due to filling up the atom table, and after a significant amount of debugging and brainstorming (and a bit of luck) we found that the culprit was this nugget in code_server: filter_mods([File|Rest], Where, Exts, Dir, Cache) -> Ext = filename:extension(File), Root = list_to_atom(filename:rootname(File, Ext)), case lists:keyfind(Ext, 2, Exts) of {Type, _} -> Key = {Type,Root}, ... This code has the effect that whenever the cache is rebuilt (happens e.g. if a requested module isn't found I believe), an atom is created (if it didn't already exist) for every file/directory in every directory of the code path. Doesn't sound too bad... - until you consider that "." is in the path by default, combined with the possibility that the emulator is started in /tmp on a busy server. The atom table was full of entries like tmp6CA6Yt tmp1R7aPK tmp8WBvWp tmpDKQd39 tmpPKCyaf tmpkijfAQ tmpofoTOG - the typical result of calls to libc mktemp(3) & friends. Obviously the atoms remained even if the files had been removed since the previous cache rebuild - it took 2-3 weeks of running time from start to crash. If the list_to_atom() call had been 2 lines further down where it should have been, the problem wouldn't have occurred... --Per Hedeland From per@REDACTED Thu Jan 26 18:05:22 2017 From: per@REDACTED (Per Hedeland) Date: Thu, 26 Jan 2017 18:05:22 +0100 Subject: [erlang-questions] Message send guarantees In-Reply-To: <20170126130940.GA41132@ferdmbp.local> References: <20170126130940.GA41132@ferdmbp.local> Message-ID: <588A2C52.2060805@hedeland.org> On 2017-01-26 14:09, Fred Hebert wrote: > On 01/25, Dmitry Kakurin wrote: >> >> The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node: >> 1. A --msg1--> C (process A sends msg1 to process C) >> 2. A --msg2--> B >> 3. B --msg2--> C (process B simply proxies msg2 from A to C) >> >> I need a guarantee that msg2 will always appear in C's queue after msg1. >> >> [...] >> >> Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do >> that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me). >> With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want. >> >> Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem? >> > > As others have mentioned, this is not really possible. You *could* observe the queue, but it is not practical. > > The interesting thing with Erlang there is that a lot of these design decisions work the same whether the processes are local or remote. The problem you see here of message ordering is very similar to > the one you get of packet ordering when delivering data over a network. > > Generally, the way to solve the problem is to label every message you send with a sequence number that increments. On every message received, the final receiver can then look at the sequence numbers > and see if any message is missing. It can then block, ask for a retransmission, or disregard duplicated messages. > > Because Erlang's guarantees for message passing only involve any given pair of processes (A -> B, or A -> C) but not more, we get a case where the following race conditions are very possible: > > A B C > | --- 1 ---> | | > | --- 2 ---------------> | > | | --- 1 --> | > > The stop at B means that through scheduling, the message can be delayed for unkown periods of time and the delays can mess up the ordering. > > The only surefire way of getting the messages to C in order is to either > a) send them directly from a single sender every time, or b) to let C know if the messages are not in order so it can do the reordering itself. Well, there is also (already hinted at I think) c) have A receive an ack from C for message 1 before it sends message 2. Doing this for every message may be an unacceptable throughput limitation in some cases, in some other cases it is what happens "anyway" due to the nature of the communication. In this particular case, "A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly", it would be sufficient that A got an ack for only the *last* via-B-message (could be a special "let's skip the proxy" message) before it started sending directly to C. Seems like a clean solution to me. --Per From per@REDACTED Thu Jan 26 18:25:33 2017 From: per@REDACTED (Per Hedeland) Date: Thu, 26 Jan 2017 18:25:33 +0100 Subject: [erlang-questions] Message send guarantees In-Reply-To: <588A2C52.2060805@hedeland.org> References: <20170126130940.GA41132@ferdmbp.local> <588A2C52.2060805@hedeland.org> Message-ID: <588A310D.2080801@hedeland.org> On 2017-01-26 18:05, Per Hedeland wrote: > On 2017-01-26 14:09, Fred Hebert wrote: >> On 01/25, Dmitry Kakurin wrote: >>> >>> The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node: >>> 1. A --msg1--> C (process A sends msg1 to process C) >>> 2. A --msg2--> B >>> 3. B --msg2--> C (process B simply proxies msg2 from A to C) >>> >>> I need a guarantee that msg2 will always appear in C's queue after msg1. >>> >>> [...] >>> >>> Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do >>> that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me). >>> With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want. Sorry, I missed that last part in my comment below - indeed it prevents the "ack solution" if C won't cooperate. --Per >>> Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem? >>> >> >> As others have mentioned, this is not really possible. You *could* observe the queue, but it is not practical. >> >> The interesting thing with Erlang there is that a lot of these design decisions work the same whether the processes are local or remote. The problem you see here of message ordering is very similar to >> the one you get of packet ordering when delivering data over a network. >> >> Generally, the way to solve the problem is to label every message you send with a sequence number that increments. On every message received, the final receiver can then look at the sequence numbers >> and see if any message is missing. It can then block, ask for a retransmission, or disregard duplicated messages. >> >> Because Erlang's guarantees for message passing only involve any given pair of processes (A -> B, or A -> C) but not more, we get a case where the following race conditions are very possible: >> >> A B C >> | --- 1 ---> | | >> | --- 2 ---------------> | >> | | --- 1 --> | >> >> The stop at B means that through scheduling, the message can be delayed for unkown periods of time and the delays can mess up the ordering. >> >> The only surefire way of getting the messages to C in order is to either >> a) send them directly from a single sender every time, or b) to let C know if the messages are not in order so it can do the reordering itself. > > Well, there is also (already hinted at I think) c) have A receive an ack > from C for message 1 before it sends message 2. Doing this for every > message may be an unacceptable throughput limitation in some cases, in > some other cases it is what happens "anyway" due to the nature of the > communication. In this particular case, "A talks to C thru proxy B and > would like to short-circuit the communication by getting rid of B and > start talking to C directly", it would be sufficient that A got an ack > for only the *last* via-B-message (could be a special "let's skip the > proxy" message) before it started sending directly to C. Seems like a > clean solution to me. > > --Per > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From mclaughlin77@REDACTED Thu Jan 26 20:27:05 2017 From: mclaughlin77@REDACTED (Kelly McLaughlin) Date: Thu, 26 Jan 2017 19:27:05 +0000 Subject: [erlang-questions] Problem building plt to dialyzer In-Reply-To: References: Message-ID: I'm having some issues that seem related to your problem. I don't have a good solution for you, but I'll add some details about my problems to the thread. I'm running Arch Linux and I I've been using erlang 18.3 from a custom build instead of the Arch package. As of about a month ago, everything was fine including dialyzer and xref. I didn't do any work with erlang for about a month and went back to it this week and found that I could no longer get dialyzer or xref to work. I get complaints about missing debug_info like this: ===> Error in dialyzing apps: Could not get abstract code for file: /home/kelly/erlang/18.3-64/lib/erlang/lib/compiler-6.0.3/ebin/beam_disasm.beam (please recompile it with +debug_info) But the module_info disagrees that debug_info is not present: 1> beam_disasm:module_info(). [{module,beam_disasm}, {exports,[{function__code,1}, {format_error,1}, {file,1}, {module_info,0}, {module_info,1}]}, {attributes,[{vsn,[161664284732293947572238212319246115161]}, {author,"Kostis Sagonas"}]}, {compile,[{options,[{outdir,"/net/isildur/ldisk/daily_build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/compiler/src/../ebin"}, {i,"/net/isildur/ldisk/daily_build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/compiler/src/../../stdlib/include"}, {i,"/net/isildur/ldisk/daily_build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/compiler/src/."}, warnings_as_errors,warn_unused_import,inline,debug_info]}, Meanwhile our build machine running Debian 8 and also using erlang 18.3 is still able to run dialyzer just fine. This makes me think it's likely an issue with a system package that may have advanced on Arch and included some changes that cause this problem. I've tried other versions of erlang as well with the same results. I unpacked some versions of 18.3 from the esl debian packages and using them got the same debug_info error. I wiped out my 18.3 PLT files trying to get things working, but I still had some 18.2.1 PLT files. I built a version of 18.2.1, but dialyzer thinks the files are invalid and if I move them out of the way then I am back to the debug_info errors. 1> dialyzer:plt_info("rebar3_18.2.1_plt"). {error,not_valid} If I copy that same PLT file to our Debian build machine and do the same the file seems fine: 1> dialyzer:plt_info("rebar3_18.2.1_plt"). {ok,[{files,["/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin/beam_a.beam", "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin/beam_asm.beam", "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin/beam_block.beam", "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin/beam_bool.beam", "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin ... Any thoughts or ideas on this are appreciated. Thanks! Hi, > > I've installed erlang 19.2 from source code on my Gentoo Linux and I'm not > able to build plt to dialyzer. It gives me the above error. The build was > configured with --without-javac --prefix=/usr > > dialyzer --build_plt --apps kernel > > dialyzer: Could not get abstract code for file: > /usr/lib/erlang/lib/kernel-5.1.1/ebin/disk_log.beam (please recompile it > with +debug_info) > > I've also tried entering on erts/emulator and running make debug and tried > to set ERL_COMPILE_FLAGS="+debug_info" with no luck. > > I don't know if it is an issue with gentoo (both my gentoo machines have > the same problem) because I have the same issue using portage and evm to > install it. > > Thank you! > > -- > Michel Almada de Castro Boaventura > Analista de Sistemas > Laborat?rio de Software Livre - LSL > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20170116/6d5b2792/attachment-0001.html > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.boaventura@REDACTED Thu Jan 26 20:33:04 2017 From: michel.boaventura@REDACTED (Michel Boaventura) Date: Thu, 26 Jan 2017 17:33:04 -0200 Subject: [erlang-questions] Problem building plt to dialyzer In-Reply-To: References: Message-ID: Hi Kelly, This turns out to be an issue with erlang and zlib > 1.2.8: http://erlang.org/pipermail/erlang-questions/2017-January/091452.html You can either downgrade zlib or apply the patch provided on the link. Cheers Em 26 de jan de 2017 17:27, "Kelly McLaughlin" escreveu: > I'm having some issues that seem related to your problem. I don't have a > good solution for you, but I'll add some details about my problems to the > thread. > > I'm running Arch Linux and I I've been using erlang 18.3 from a custom > build instead of the Arch package. As of about a month ago, everything was > fine including dialyzer and xref. I didn't do any work with erlang for > about a month and went back to it this week and found that I could no > longer get dialyzer or xref to work. I get complaints about missing > debug_info like this: > > ===> Error in dialyzing apps: Could not get abstract code for file: > /home/kelly/erlang/18.3-64/lib/erlang/lib/compiler-6.0.3/ebin/beam_disasm.beam > (please recompile it with +debug_info) > > But the module_info disagrees that debug_info is not present: > > 1> beam_disasm:module_info(). > [{module,beam_disasm}, > {exports,[{function__code,1}, > {format_error,1}, > {file,1}, > {module_info,0}, > {module_info,1}]}, > {attributes,[{vsn,[161664284732293947572238212319246115161]}, > {author,"Kostis Sagonas"}]}, > {compile,[{options,[{outdir,"/net/isildur/ldisk/daily_ > build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/ > compiler/src/../ebin"}, > {i,"/net/isildur/ldisk/daily_ > build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/ > compiler/src/../../stdlib/include"}, > {i,"/net/isildur/ldisk/daily_ > build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/compiler/src/."}, > warnings_as_errors,warn_unused_import,inline,debug_ > info]}, > > Meanwhile our build machine running Debian 8 and also using erlang 18.3 is > still able to run dialyzer just fine. This makes me think it's likely an > issue with a system package that may have advanced on Arch and included > some changes that cause this problem. > > I've tried other versions of erlang as well with the same results. I > unpacked some versions of 18.3 from the esl debian packages and using them > got the same debug_info error. I wiped out my 18.3 PLT files trying to get > things working, but I still had some 18.2.1 PLT files. I built a version of > 18.2.1, but dialyzer thinks the files are invalid and if I move them out of > the way then I am back to the debug_info errors. > > 1> dialyzer:plt_info("rebar3_18.2.1_plt"). > {error,not_valid} > > If I copy that same PLT file to our Debian build machine and do the same > the file seems fine: > > 1> dialyzer:plt_info("rebar3_18.2.1_plt"). > {ok,[{files,["/home/kelly/erlang/18.2.1-64/lib/erlang/ > lib/compiler-6.0.2/ebin/beam_a.beam", > "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ > ebin/beam_asm.beam", > "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ > ebin/beam_block.beam", > "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ > ebin/beam_bool.beam", > "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ > ebin > ... > > Any thoughts or ideas on this are appreciated. Thanks! > > > Hi, >> >> I've installed erlang 19.2 from source code on my Gentoo Linux and I'm not >> able to build plt to dialyzer. It gives me the above error. The build was >> configured with --without-javac --prefix=/usr >> >> dialyzer --build_plt --apps kernel >> >> dialyzer: Could not get abstract code for file: >> /usr/lib/erlang/lib/kernel-5.1.1/ebin/disk_log.beam (please recompile it >> with +debug_info) >> >> I've also tried entering on erts/emulator and running make debug and tried >> to set ERL_COMPILE_FLAGS="+debug_info" with no luck. >> >> I don't know if it is an issue with gentoo (both my gentoo machines have >> the same problem) because I have the same issue using portage and evm to >> install it. >> >> Thank you! >> >> -- >> Michel Almada de Castro Boaventura >> Analista de Sistemas >> Laborat?rio de Software Livre - LSL >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: > 20170116/6d5b2792/attachment-0001.html> >> >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mclaughlin77@REDACTED Thu Jan 26 20:39:34 2017 From: mclaughlin77@REDACTED (Kelly McLaughlin) Date: Thu, 26 Jan 2017 19:39:34 +0000 Subject: [erlang-questions] Problem building plt to dialyzer In-Reply-To: References: Message-ID: Worked like a charm! Thanks for the tip, Michel. On Thu, Jan 26, 2017 at 12:33 PM Michel Boaventura < michel.boaventura@REDACTED> wrote: > Hi Kelly, > > This turns out to be an issue with erlang and zlib > 1.2.8: > > http://erlang.org/pipermail/erlang-questions/2017-January/091452.html > > You can either downgrade zlib or apply the patch provided on the link. > > Cheers > > Em 26 de jan de 2017 17:27, "Kelly McLaughlin" > escreveu: > > I'm having some issues that seem related to your problem. I don't have a > good solution for you, but I'll add some details about my problems to the > thread. > > I'm running Arch Linux and I I've been using erlang 18.3 from a custom > build instead of the Arch package. As of about a month ago, everything was > fine including dialyzer and xref. I didn't do any work with erlang for > about a month and went back to it this week and found that I could no > longer get dialyzer or xref to work. I get complaints about missing > debug_info like this: > > ===> Error in dialyzing apps: Could not get abstract code for file: > /home/kelly/erlang/18.3-64/lib/erlang/lib/compiler-6.0.3/ebin/beam_disasm.beam > (please recompile it with +debug_info) > > But the module_info disagrees that debug_info is not present: > > 1> beam_disasm:module_info(). > [{module,beam_disasm}, > {exports,[{function__code,1}, > {format_error,1}, > {file,1}, > {module_info,0}, > {module_info,1}]}, > {attributes,[{vsn,[161664284732293947572238212319246115161]}, > {author,"Kostis Sagonas"}]}, > > {compile,[{options,[{outdir,"/net/isildur/ldisk/daily_build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/compiler/src/../ebin"}, > > {i,"/net/isildur/ldisk/daily_build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/compiler/src/../../stdlib/include"}, > > {i,"/net/isildur/ldisk/daily_build/18_prebuild_opu_o.2016-03-14_21/otp_src_18/lib/compiler/src/."}, > > warnings_as_errors,warn_unused_import,inline,debug_info]}, > > Meanwhile our build machine running Debian 8 and also using erlang 18.3 is > still able to run dialyzer just fine. This makes me think it's likely an > issue with a system package that may have advanced on Arch and included > some changes that cause this problem. > > I've tried other versions of erlang as well with the same results. I > unpacked some versions of 18.3 from the esl debian packages and using them > got the same debug_info error. I wiped out my 18.3 PLT files trying to get > things working, but I still had some 18.2.1 PLT files. I built a version of > 18.2.1, but dialyzer thinks the files are invalid and if I move them out of > the way then I am back to the debug_info errors. > > 1> dialyzer:plt_info("rebar3_18.2.1_plt"). > {error,not_valid} > > If I copy that same PLT file to our Debian build machine and do the same > the file seems fine: > > 1> dialyzer:plt_info("rebar3_18.2.1_plt"). > > {ok,[{files,["/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin/beam_a.beam", > > "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin/beam_asm.beam", > > "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin/beam_block.beam", > > "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin/beam_bool.beam", > > "/home/kelly/erlang/18.2.1-64/lib/erlang/lib/compiler-6.0.2/ebin > ... > > Any thoughts or ideas on this are appreciated. Thanks! > > > Hi, > > I've installed erlang 19.2 from source code on my Gentoo Linux and I'm not > able to build plt to dialyzer. It gives me the above error. The build was > configured with --without-javac --prefix=/usr > > dialyzer --build_plt --apps kernel > > dialyzer: Could not get abstract code for file: > /usr/lib/erlang/lib/kernel-5.1.1/ebin/disk_log.beam (please recompile it > with +debug_info) > > I've also tried entering on erts/emulator and running make debug and tried > to set ERL_COMPILE_FLAGS="+debug_info" with no luck. > > I don't know if it is an issue with gentoo (both my gentoo machines have > the same problem) because I have the same issue using portage and evm to > install it. > > Thank you! > > -- > Michel Almada de Castro Boaventura > Analista de Sistemas > Laborat?rio de Software Livre - LSL > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20170116/6d5b2792/attachment-0001.html > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cean.ebengt@REDACTED Thu Jan 26 21:09:10 2017 From: cean.ebengt@REDACTED (bengt) Date: Thu, 26 Jan 2017 21:09:10 +0100 Subject: [erlang-questions] Looking for Erlang code that handles PDF file attachments In-Reply-To: References: Message-ID: <21810F27-11CF-4FCC-9BD2-A56D1CA0E799@gmail.com> Thank you for the tip. When I look at the Erlguten documentation it seems to write-only of PDF. No read functions. bengt > On 26 Jan 2017, at 10:28, Chandru wrote: > > Have you looked into erlguten? > > https://github.com/richcarl/erlguten > > cheers, > Chandru > > On 25 January 2017 at 21:06, bengt > wrote: > Greetings, > > I would like to attach files (or some sort of binary content, anyway) to PDF files, without destroying the PDF'ness of the file. I would also like to be able to read the created PDF file and be able to tell what is the attachment (and what is the original content). I have found Itext (Java, see http://itextpdf.com/ ). It seems to be able to do that and everything else too, but is there any other alternative? Preferably in Erlang. > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dima_kakurin@REDACTED Fri Jan 27 03:25:49 2017 From: dima_kakurin@REDACTED (Dmitry Kakurin) Date: Fri, 27 Jan 2017 02:25:49 +0000 Subject: [erlang-questions] Message send guarantees In-Reply-To: References: , Message-ID: From: Joe Armstrong Sent: Thursday, January 26, 2017 5:29 AM On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin wrote: > When I execute "pid ! msg" and it returns, what are the guarantees if pid > and self are on the same node? > Can I assume that by the time it returns, msg is in the pid's queue? And as > a result all messages sent to pid afterwards will be queued after msg? > > The reason I'm asking is because I need a guarantee of proper message > ordering in the following scenario when all processes involved are local to > a node: > 1. A --msg1--> C (process A sends msg1 to process C) > 2. A --msg2--> B > 3. B --msg2--> C (process B simply proxies msg2 from A to C) > > I need a guarantee that msg2 will always appear in C's queue after msg1. This cannot be guaranteed Thanks Joe and others. I understand now that it's fruitless to pursue a direction where I'd try to somehow artificially establish this guarantee. I have to re-evaluate my constraints then. All along my main constraint was that C can be any gen_server, it's not owned by me and I cannot change it. If I could change C then the following would work: 1. A --msg1--> C 2. A --ping--> C 3. A <--pong-- C 4. A --msg2--> B 5. B --msg2--> C All we need from C is to respond "pong" to "ping" message. In this case A knows msg2 is enqueued after msg1 because A has observed "pong" in response to "ping" that was sent after msg1. The ordering is as desired here. Now the only problem I have to solve is: what can be used as a benign message that I can send to any gen_server such that it would reply (to emulate ping/pong exchange above)? Looking thru gen_server source code I see that it's processing sys messages. So I can use get_state message as "ping" and response with state as "pong". Except it may not be really benign perf-wise if amount of state kept by C is significant. There is also get_status, but I don't understand how it's processed and if I can use it for ping/pong purposes. Can I? Is there a better message? Do you have a better idea for ping/pong messages on C? Or how to solve this in general? Thank you, Dmitry. P.S. I know that in this simplified example I could piggyback on msg1 if it returns something but in real code I cannot, so let's assume it's a one-way message (a cast, not a call). So I'm OK with ping/pong. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Fri Jan 27 08:41:32 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 27 Jan 2017 08:41:32 +0100 Subject: [erlang-questions] Message send guarantees In-Reply-To: References: Message-ID: <20170127074132.GA6098@erix.ericsson.se> On Fri, Jan 27, 2017 at 02:25:49AM +0000, Dmitry Kakurin wrote: > From: Joe Armstrong > > Sent: Thursday, January 26, 2017 5:29 AM > > On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin > wrote: > > When I execute "pid ! msg" and it returns, what are the guarantees if pid > > and self are on the same node? > > Can I assume that by the time it returns, msg is in the pid's queue? And as > > a result all messages sent to pid afterwards will be queued after msg? > > > > The reason I'm asking is because I need a guarantee of proper message > > ordering in the following scenario when all processes involved are local to > > a node: > > 1. A --msg1--> C (process A sends msg1 to process C) > > 2. A --msg2--> B > > 3. B --msg2--> C (process B simply proxies msg2 from A to C) > > > > I need a guarantee that msg2 will always appear in C's queue after msg1. > > This cannot be guaranteed > > Thanks Joe and others. I understand now that it's fruitless to pursue a direction where I'd try to somehow artificially establish this guarantee. > > I have to re-evaluate my constraints then. All along my main constraint was that C can be any gen_server, it's not owned by me and I cannot change it. > If I could change C then the following would work: > 1. A --msg1--> C > 2. A --ping--> C > 3. A <--pong-- C > 4. A --msg2--> B > 5. B --msg2--> C > Have you changed your question? Previously the problem was to send non-proxied messages after proxied messages. But if B is a message relay (receive Msg -> C ! Msg, loop() end), and provided the ping/pong is a gen_server call (since the request contains who should have the reply, then this should also work: 1. A --msg1---> B 2. A --req----> B 3. B --msg1---> C 4. B --req----> C 5. A <--reply-- C 6. A --msg2---> C > All we need from C is to respond "pong" to "ping" message. In this case A knows msg2 is enqueued after msg1 because A has observed "pong" in response to "ping" that was sent after msg1. The ordering is as desired here. > Now the only problem I have to solve is: what can be used as a benign message that I can send to any gen_server such that it would reply (to emulate ping/pong exchange above)? > Looking thru gen_server source code I see that it's processing sys messages. So I can use get_state message as "ping" and response with state as "pong". Except it may not be really benign perf-wise if amount of state kept by C is significant. > There is also get_status, but I don't understand how it's processed and if I can use it for ping/pong purposes. Can I? Is there a better message? sys:get_status/1,2 returns much more data including what sys:get_state/1,2 returns, so it is less suitable. If you know anything more about your gen_server C, it may have some status quering handle_call that can be used. Maybe sys:statistics/2 can be good enough. Mostly no statistics collection is activated for a given gen_server so you will get just {ok,no_statistics} back as pong, and if statistics collection should be activated the gen_server does not much more than process_info(self(), reductions) and erlang:localtime(), which may and may not be regarded as heavy, but compared to returning the whole arbitrary gen_server state it may be better. > > Do you have a better idea for ping/pong messages on C? Or how to solve this in general? > > Thank you, Dmitry. > > P.S. I know that in this simplified example I could piggyback on msg1 if it returns something but in real code I cannot, so let's assume it's a one-way message (a cast, not a call). So I'm OK with ping/pong. > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From andrey@REDACTED Fri Jan 27 11:46:53 2017 From: andrey@REDACTED (Andrey Tsirulev) Date: Fri, 27 Jan 2017 13:46:53 +0300 Subject: [erlang-questions] Message send guarantees In-Reply-To: <20170126071421.GA46222@erix.ericsson.se> References: <20170126071421.GA46222@erix.ericsson.se> Message-ID: <36c826c2-0c84-4370-9896-4299cb09d5ad@artplant.no> Hello, 26.01.2017 10:14, Raimo Niskanen wrote: > On Wed, Jan 25, 2017 at 11:06:05PM +0000, Dmitry Kakurin wrote: >> No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow. >> >> What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why. > On a multi-threade system it is very hard to know in which order things > happen, so there is no "reordering" since there is no temporal order > to begin with. What about the following example? A -- msg1 ---> A A -- msg2 ---> B B -- msg2 ---> A In this case, is it guaranteed that if both msg2 and msg1 get to A's msg queue, msg1 gets there earlier? If this is guaranteed, what makes this example *really* different from the original example? I use this pattern for delayed gen_server's initialization if it's time consuming, like that: init(Args) -> self() ! {init, Args}, ... handle_info({init, Args}, State) -> %% time consuming initialization here to return from init ASAP so that start/start_link do not block for a long time. Then I expect that after P = gen_server:start_link(...), P ! test when test is received, gen_server is fully initialized. It never caused problems, but it might be luck. Best Regards, Andrey From michael.c.williams10@REDACTED Fri Jan 27 14:57:37 2017 From: michael.c.williams10@REDACTED (Mike Williams) Date: Fri, 27 Jan 2017 13:57:37 +0000 Subject: [erlang-questions] [Erlang-JIRA] (ERL-345) Cannot compile NIFs for modules with dots in name In-Reply-To: References: Message-ID: Forwarding mail. On 27 Jan 2017 1:55 p.m., "ukasiu (JIRA)" wrote: > ukasiu created ERL-345: > -------------------------- > > Summary: Cannot compile NIFs for modules with dots in name > Key: ERL-345 > URL: https://bugs.erlang.org/browse/ERL-345 > Project: Erlang/OTP > Issue Type: Bug > Reporter: ukasiu > Attachments: fast_compare.c > > I'm trying to compile NIF with {{STATIC_ERLANG_NIF}} to use in Elixir. > {{ERL_NIF_INIT}} looks like this {{ERL_NIF_INIT(Elixir.FastCompare, > nif_funcs, NULL, NULL, NULL, NULL)}} -- {{MODNAME}} contains dot. I'm > getting this when trying to compile: > > {noformat} > In file included from fast_compare.c:2:0: > fast_compare.c:20:20: error: expected '=', ',', ';', 'asm' or > '__attribute__' before '.' token > ERL_NIF_INIT(Elixir.FastCompare, nif_funcs, NULL, NULL, NULL, NULL) > ^ > .../erts-8.2.1/include/erl_nif.h:264:51: note: in definition of macro > 'ERL_NIF_INIT_DECL' > # define ERL_NIF_INIT_DECL(MODNAME) ErlNifEntry* MODNAME ## > _nif_init(ERL_NIF_INIT_ARGS) > ^ > fast_compare.c:20:1: note: in expansion of macro 'ERL_NIF_INIT' > ERL_NIF_INIT(Elixir.FastCompare, nif_funcs, NULL, NULL, NULL, NULL) > ^ > fast_compare.c:20:20: error: expected '=', ',', ';', 'asm' or > '__attribute__' before '.' token > ERL_NIF_INIT(Elixir.FastCompare, nif_funcs, NULL, NULL, NULL, NULL) > ^ > .../erts-8.2.1/include/erl_nif.h:264:51: note: in definition of macro > 'ERL_NIF_INIT_DECL' > # define ERL_NIF_INIT_DECL(MODNAME) ErlNifEntry* MODNAME ## > _nif_init(ERL_NIF_INIT_ARGS) > ^ > fast_compare.c:20:1: note: in expansion of macro 'ERL_NIF_INIT' > ERL_NIF_INIT(Elixir.FastCompare, nif_funcs, NULL, NULL, NULL, NULL) > ^ > {noformat} > > > > > -- > This message was sent by Atlassian JIRA > (v6.4.7#64022) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From valerii.tikhonov@REDACTED Fri Jan 27 18:53:39 2017 From: valerii.tikhonov@REDACTED (Valery Tikhonov) Date: Fri, 27 Jan 2017 18:53:39 +0100 Subject: [erlang-questions] Meet seagull - new erlang library for service location, configuration sharing and working with consul/etcd Message-ID: Hej! I have created a new library, that can be useful. A client to consul/etcd for working with shared configuration and service location. I am using it in microservices, feel free to try in yours. Regards, Val -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Fri Jan 27 20:22:33 2017 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 27 Jan 2017 19:22:33 +0000 Subject: [erlang-questions] Meet seagull - new erlang library for service location, configuration sharing and working with consul/etcd In-Reply-To: References: Message-ID: link? On Fri, 27 Jan 2017 at 20:14, Valery Tikhonov wrote: > Hej! > I have created a new library, that can be useful. A client to consul/etcd > for working with shared configuration and service location. > I am using it in microservices, feel free to try in yours. > Regards, Val > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From valerii.tikhonov@REDACTED Fri Jan 27 20:22:55 2017 From: valerii.tikhonov@REDACTED (Valery Tikhonov) Date: Fri, 27 Jan 2017 20:22:55 +0100 Subject: [erlang-questions] Meet seagull - new erlang library for service location, configuration sharing and working with consul/etcd Message-ID: ups. Friday evening) Here is the link - https://github.com/SeaBattle/seagull -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnson786.kevin@REDACTED Sat Jan 28 01:55:51 2017 From: johnson786.kevin@REDACTED (Kevin Johnson) Date: Fri, 27 Jan 2017 19:55:51 -0500 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term Message-ID: Hi, I am using Elixir to send data using :gen_tcp. The syntax presented here may be elementary Elixir syntax, however the issue is specifically related to the usage of term_to_binary and binary_to_term conversions over gen_tcp. This is the command I used on the client side: > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) This is the command I use on the server side: > defp handle_data(data) do > data > |> :erlang.binary_to_term > end When my data gets received on the server side, it seems that at a certain size threshold that it errors out like this: pry(6)> [error] GenServer #PID<0.502.0> terminating ** (ArgumentError) argument error :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, 49, > 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101, > 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, ...>>) (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, > 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, > 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, > ...>>} Here is some example data that will just pass fine: %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"choice_id" => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"choice_id" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => "B5_B2_Q5_C1"}, %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"choice_id" => "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => "B3_B4_Q1_C1"}], "B2_Q16" => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => [%{"choice_id" => "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], "B2_Q9" => [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => "B2_Q17_C1"}], "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], "B2_Q4" => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" => "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" => [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => "B2_Q2_C1"}], "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} The moment I make it slightly longer, by like appending an extra key "a" => "b" it will give me the above error. It seems to be that the size of the keys plays an issue here, because if I convert all the above "choice_id" keys to a mere "c", then even with the extra entry "a" => "b" included like following below will work just fine: %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" => "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" => "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => [%{"c" => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => [%{"c" => "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => [%{"c" => "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => [%{"c" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => [%{"c" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"c" => "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], "B2_Q10" => [%{"c" => "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, %{"c" => "B4_Q1_C10"}], "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => [%{"c" => "B2_Q6_C1"}], "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => [%{"c" => "B2_Q1_C1"}], "a" => "b"} The following on the other hand will just fail: data = %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" => "q", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" => "q", "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" => "q"} :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) The only issue that I was able to track down here is a conjunction of two things: 1) A limit to the size of total key length inside a map/hash 2) sending that hash as a binary over tcp and converting back to term In all of the above cases, a direct conversion of :erlang.term_to_binary followed by :erlang.binary_to_term works just fine without any issues. The issue only comes about after that binary was send over tcp and then :erlang.binary_to_term is attempted. I would greatly appreciate if anyone can provide me guidance in this matter, point out to me any relevant documentation that officially states any of these constraints if they are in fact constraints, and if they are not supposed to be constraints then kindly instruct me what additional information may be needed for me to get to the bottom of this. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Sat Jan 28 08:12:07 2017 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Sat, 28 Jan 2017 08:12:07 +0100 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term In-Reply-To: References: Message-ID: What is your zlib version? There has been a thread about a very similar issue and it was pinpointed to be related to the zlib version in your system: http://erlang.org/pipermail/erlang-questions/2017-January/091434.html *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson wrote: > Hi, > > I am using Elixir to send data using :gen_tcp. The syntax presented here > may be elementary Elixir syntax, however the issue is specifically related > to the usage of term_to_binary and binary_to_term conversions over gen_tcp. > > This is the command I used on the client side: > >> :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > > This is the command I use on the server side: > >> defp handle_data(data) do >> data >> |> :erlang.binary_to_term >> end > > > When my data gets received on the server side, it seems that at a certain > size threshold that it errors out like this: > > pry(6)> [error] GenServer #PID<0.502.0> terminating > > ** (ArgumentError) argument error > > :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, 49, >> 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101, >> 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, ...>>) > > (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 > > (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 > > (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 > > Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0, >> 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, >> 4, 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, >> ...>>} > > > Here is some example data that will just pass fine: > %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => > [%{"choice_id" => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => > "B2_Q7_C1"}], "B5_B1_Q8" => [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" > => [%{"choice_id" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => > "B5_B2_Q5_C1"}, %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => > [%{"choice_id" => "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => > "B3_B4_Q1_C1"}], "B2_Q16" => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => > [%{"choice_id" => "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], > "B2_Q9" => [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => > "B2_Q17_C1"}], "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" > => [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => > "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], "B2_Q4" > => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => > "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" => > "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" => > [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => "B2_Q2_C1"}], > "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} > > The moment I make it slightly longer, by like appending an extra key "a" > => "b" it will give me the above error. > > It seems to be that the size of the keys plays an issue here, because if I > convert all the above "choice_id" keys to a mere "c", then even with the > extra entry "a" => "b" included like following below will work just fine: > %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => > "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" => > "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => > [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" => > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => > [%{"c" => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => > [%{"c" => "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => > [%{"c" => "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], > "B5_B2_Q3" => [%{"c" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > [%{"c" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], > "B5_B2_Q2" => [%{"c" => "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], > "B2_Q10" => [%{"c" => "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, > %{"c" => "B4_Q1_C10"}], "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => > [%{"c" => "B2_Q6_C1"}], "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => > [%{"c" => "B2_Q1_C1"}], "a" => "b"} > > The following on the other hand will just fail: > data = %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" => "q", " > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" => "q", " > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" > => "q"} > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > The only issue that I was able to track down here is a conjunction of two > things: > 1) A limit to the size of total key length inside a map/hash > 2) sending that hash as a binary over tcp and converting back to term > > In all of the above cases, a direct conversion of :erlang.term_to_binary > followed by :erlang.binary_to_term works just fine without any issues. The > issue only comes about after that binary was send over tcp and then > :erlang.binary_to_term is attempted. > > I would greatly appreciate if anyone can provide me guidance in this > matter, point out to me any relevant documentation that officially states > any of these constraints if they are in fact constraints, and if they are > not supposed to be constraints then kindly instruct me what additional > information may be needed for me to get to the bottom of this. > > Thank you. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dszoboszlay@REDACTED Sat Jan 28 09:19:19 2017 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Sat, 28 Jan 2017 08:19:19 +0000 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term In-Reply-To: References: Message-ID: I don't think this would be zlib related. TCP is simply a stream protocol: you put in a stream of bytes in arbitrary chunks, and on the other end you receive the same sequence of bytes, but possibly in different chunks. If you want to send a sequence of messages, it is up to the application layer to detect the message boundaries. You can use e.g. the {packet, 4} option when opening the TCP socket (on both ends), so Erlang will add a 4-byte size header to every message you send, and on the receiving end it will wait for a whole message to arrive before handing off the bytes to your application. This will work only up to 4GB messages of course. Alternatively, you may implement a streaming version of binary_to_term, which will tell you if you gave it too few or too many bytes to parse a whole term from them. Cheers, Daniel On Sat, 28 Jan 2017 at 08:12 Jos? Valim wrote: > What is your zlib version? There has been a thread about a very similar > issue and it was pinpointed to be related to the zlib version in your > system: > http://erlang.org/pipermail/erlang-questions/2017-January/091434.html > > > > *Jos? Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Director of R&D > > On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson > wrote: > > Hi, > > I am using Elixir to send data using :gen_tcp. The syntax presented here > may be elementary Elixir syntax, however the issue is specifically related > to the usage of term_to_binary and binary_to_term conversions over gen_tcp. > > This is the command I used on the client side: > > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > > This is the command I use on the server side: > > defp handle_data(data) do > data > |> :erlang.binary_to_term > end > > > When my data gets received on the server side, it seems that at a certain > size threshold that it errors out like this: > > pry(6)> [error] GenServer #PID<0.502.0> terminating > > ** (ArgumentError) argument error > > :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, 49, > 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101, > 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, ...>>) > > (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 > > (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 > > (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 > > Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0, > 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, > 4, 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, > ...>>} > > > Here is some example data that will just pass fine: > %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => > [%{"choice_id" => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => > "B2_Q7_C1"}], "B5_B1_Q8" => [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" > => [%{"choice_id" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => > "B5_B2_Q5_C1"}, %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => > [%{"choice_id" => "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => > "B3_B4_Q1_C1"}], "B2_Q16" => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => > [%{"choice_id" => "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], > "B2_Q9" => [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => > "B2_Q17_C1"}], "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" > => [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => > "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], "B2_Q4" > => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => > "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" => > "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" => > [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => "B2_Q2_C1"}], > "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} > > The moment I make it slightly longer, by like appending an extra key "a" > => "b" it will give me the above error. > > It seems to be that the size of the keys plays an issue here, because if I > convert all the above "choice_id" keys to a mere "c", then even with the > extra entry "a" => "b" included like following below will work just fine: > %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => > "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" => > "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => > [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" => > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => > [%{"c" => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => > [%{"c" => "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => > [%{"c" => "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], > "B5_B2_Q3" => [%{"c" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > [%{"c" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], > "B5_B2_Q2" => [%{"c" => "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], > "B2_Q10" => [%{"c" => "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, > %{"c" => "B4_Q1_C10"}], "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => > [%{"c" => "B2_Q6_C1"}], "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => > [%{"c" => "B2_Q1_C1"}], "a" => "b"} > > The following on the other hand will just fail: > data = > %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" > => "q", > "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" > => "q", > "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" > => "q"} > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > The only issue that I was able to track down here is a conjunction of two > things: > 1) A limit to the size of total key length inside a map/hash > 2) sending that hash as a binary over tcp and converting back to term > > In all of the above cases, a direct conversion of :erlang.term_to_binary > followed by :erlang.binary_to_term works just fine without any issues. The > issue only comes about after that binary was send over tcp and then > :erlang.binary_to_term is attempted. > > I would greatly appreciate if anyone can provide me guidance in this > matter, point out to me any relevant documentation that officially states > any of these constraints if they are in fact constraints, and if they are > not supposed to be constraints then kindly instruct me what additional > information may be needed for me to get to the bottom of this. > > Thank you. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thom.cherryhomes@REDACTED Sat Jan 28 10:08:06 2017 From: thom.cherryhomes@REDACTED (Thom Cherryhomes) Date: Sat, 28 Jan 2017 09:08:06 +0000 Subject: [erlang-questions] Properly decoding telnet IAC negotiations. Message-ID: Another round of late late night hacking. I've been further experimenting with ranch to implement a proper Telnet implementation (for those who thought Telnet was simply a TCP connection from A to B...nope..) First, I wanted to compliment the softwre engineers who worked on gen_statem, it's a fantastic replacement for gen_fsm which consolidates a lot of previously seperate functions into something that is simpler to implement, but has a lot more functionality. As part of the Telnet negotiation process, I am having to send a string of IAC codes to specify no echo, character mode, etc. to get the terminal characteristics that I want: negotiation(internal, preamble, #data{transport=Transport,socket=Socket}=Data) -> io:format("Sending preamble~n"), Transport:send(Socket,<<255,253,1,255,253,31,255,251,1,255,251,3>>), {next_state, negotiation, Data}; but now that I am sending this data, I am getting the terminal characteristics that I want, but as part of the negotiation, the client not only acknowledges the attributes that I wanted, but the client responds with attributes of its own: 1> EventType info -- Msg {tcp,#Port<0.638>, <<255,252,1,255,251,31,255,250,31,0,170,0,113,255, 240,255,253,1,255,253,3>>} -- Data {data, nakbbs_telnet_ascii, #Port<0.638>, ranch_tcp} -- I know how to parse each of these bits, but am wondering how best to do the appropriate pattern matching to deal with each incoming bit of info. The one saving grace is that each piece of the reply is dileniated with 255 followed by the protocol option to change (1 = echo, 31 = window size, etc.) Is the correct approach to snip off each dileniated piece from the binary and match? -Thom -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.boaventura@REDACTED Sat Jan 28 13:42:45 2017 From: michel.boaventura@REDACTED (Michel Boaventura) Date: Sat, 28 Jan 2017 10:42:45 -0200 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term In-Reply-To: References: Message-ID: I do think it's related with zlib, since he is calling term_to_binary, which uses it by default. Em 28 de jan de 2017 06:19, "D?niel Szoboszlay" escreveu: I don't think this would be zlib related. TCP is simply a stream protocol: you put in a stream of bytes in arbitrary chunks, and on the other end you receive the same sequence of bytes, but possibly in different chunks. If you want to send a sequence of messages, it is up to the application layer to detect the message boundaries. You can use e.g. the {packet, 4} option when opening the TCP socket (on both ends), so Erlang will add a 4-byte size header to every message you send, and on the receiving end it will wait for a whole message to arrive before handing off the bytes to your application. This will work only up to 4GB messages of course. Alternatively, you may implement a streaming version of binary_to_term, which will tell you if you gave it too few or too many bytes to parse a whole term from them. Cheers, Daniel On Sat, 28 Jan 2017 at 08:12 Jos? Valim wrote: > What is your zlib version? There has been a thread about a very similar > issue and it was pinpointed to be related to the zlib version in your > system: http://erlang.org/pipermail/erlang-questions/ > 2017-January/091434.html > > > > *Jos? Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Director of R&D > > On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson > wrote: > > Hi, > > I am using Elixir to send data using :gen_tcp. The syntax presented here > may be elementary Elixir syntax, however the issue is specifically related > to the usage of term_to_binary and binary_to_term conversions over gen_tcp. > > This is the command I used on the client side: > > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > > This is the command I use on the server side: > > defp handle_data(data) do > data > |> :erlang.binary_to_term > end > > > When my data gets received on the server side, it seems that at a certain > size threshold that it errors out like this: > > pry(6)> [error] GenServer #PID<0.502.0> terminating > > ** (ArgumentError) argument error > > :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, 49, > 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101, > 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, ...>>) > > (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 > > (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 > > (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 > > Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0, > 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, > 4, 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, > ...>>} > > > Here is some example data that will just pass fine: > %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => > [%{"choice_id" => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => > "B2_Q7_C1"}], "B5_B1_Q8" => [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" > => [%{"choice_id" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => > "B5_B2_Q5_C1"}, %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => > [%{"choice_id" => "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => > "B3_B4_Q1_C1"}], "B2_Q16" => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => > [%{"choice_id" => "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], > "B2_Q9" => [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => > "B2_Q17_C1"}], "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" > => [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => > "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], "B2_Q4" > => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => > "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" => > "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" => > [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => "B2_Q2_C1"}], > "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} > > The moment I make it slightly longer, by like appending an extra key "a" > => "b" it will give me the above error. > > It seems to be that the size of the keys plays an issue here, because if I > convert all the above "choice_id" keys to a mere "c", then even with the > extra entry "a" => "b" included like following below will work just fine: > %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => > "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" => > "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => > [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" => > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => > [%{"c" => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => > [%{"c" => "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => > [%{"c" => "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], > "B5_B2_Q3" => [%{"c" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > [%{"c" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], > "B5_B2_Q2" => [%{"c" => "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], > "B2_Q10" => [%{"c" => "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, > %{"c" => "B4_Q1_C10"}], "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => > [%{"c" => "B2_Q6_C1"}], "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => > [%{"c" => "B2_Q1_C1"}], "a" => "b"} > > The following on the other hand will just fail: > data = %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" => "q", " > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" => "q", " > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" > => "q"} > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > The only issue that I was able to track down here is a conjunction of two > things: > 1) A limit to the size of total key length inside a map/hash > 2) sending that hash as a binary over tcp and converting back to term > > In all of the above cases, a direct conversion of :erlang.term_to_binary > followed by :erlang.binary_to_term works just fine without any issues. The > issue only comes about after that binary was send over tcp and then > :erlang.binary_to_term is attempted. > > I would greatly appreciate if anyone can provide me guidance in this > matter, point out to me any relevant documentation that officially states > any of these constraints if they are in fact constraints, and if they are > not supposed to be constraints then kindly instruct me what additional > information may be needed for me to get to the bottom of this. > > Thank you. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From james@REDACTED Sat Jan 28 13:52:45 2017 From: james@REDACTED (James Fish) Date: Sat, 28 Jan 2017 12:52:45 +0000 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term In-Reply-To: References: Message-ID: term_to_binary only uses zlib compression when the :compressed option is used, and then it would start with <<131, 80, ...>>. The binary starts with <<131, 116, ...>>, which is an uncompressed map in external term format (as expected). I think Daniel gave the correct suggestion, especially if the cutoff is 1460 bytes. This is the default internal buffer size for gen_tcp sockets and so the maximum number of bytes received using an active socket with `packet: :raw` (the default). On 28 January 2017 at 12:42, Michel Boaventura wrote: > I do think it's related with zlib, since he is calling term_to_binary, > which uses it by default. > > > Em 28 de jan de 2017 06:19, "D?niel Szoboszlay" > escreveu: > > I don't think this would be zlib related. TCP is simply a stream protocol: > you put in a stream of bytes in arbitrary chunks, and on the other end you > receive the same sequence of bytes, but possibly in different chunks. If > you want to send a sequence of messages, it is up to the application layer > to detect the message boundaries. > > You can use e.g. the {packet, 4} option when opening the TCP socket (on > both ends), so Erlang will add a 4-byte size header to every message you > send, and on the receiving end it will wait for a whole message to arrive > before handing off the bytes to your application. This will work only up to > 4GB messages of course. > > Alternatively, you may implement a streaming version of binary_to_term, > which will tell you if you gave it too few or too many bytes to parse a > whole term from them. > > Cheers, > Daniel > > On Sat, 28 Jan 2017 at 08:12 Jos? Valim > wrote: > >> What is your zlib version? There has been a thread about a very similar >> issue and it was pinpointed to be related to the zlib version in your >> system: http://erlang.org/pipermail/erlang-questions/2017- >> January/091434.html >> >> >> >> *Jos? Valim* >> www.plataformatec.com.br >> Skype: jv.ptec >> Founder and Director of R&D >> >> On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson < >> johnson786.kevin@REDACTED> wrote: >> >> Hi, >> >> I am using Elixir to send data using :gen_tcp. The syntax presented here >> may be elementary Elixir syntax, however the issue is specifically related >> to the usage of term_to_binary and binary_to_term conversions over gen_tcp. >> >> This is the command I used on the client side: >> >> :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) >> >> >> This is the command I use on the server side: >> >> defp handle_data(data) do >> data >> |> :erlang.binary_to_term >> end >> >> >> When my data gets received on the server side, it seems that at a certain >> size threshold that it errors out like this: >> >> pry(6)> [error] GenServer #PID<0.502.0> terminating >> >> ** (ArgumentError) argument error >> >> :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, 49, >> 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101, >> 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, ...>>) >> >> (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 >> >> (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 >> >> (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 >> >> Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0, >> 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, >> 4, 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, >> ...>>} >> >> >> Here is some example data that will just pass fine: >> %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => >> [%{"choice_id" => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => >> "B2_Q7_C1"}], "B5_B1_Q8" => [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" >> => [%{"choice_id" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => >> "B5_B2_Q5_C1"}, %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => >> [%{"choice_id" => "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => >> "B3_B4_Q1_C1"}], "B2_Q16" => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => >> [%{"choice_id" => "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], >> "B2_Q9" => [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => >> "B2_Q17_C1"}], "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" >> => [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => >> "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => >> [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => >> "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], "B2_Q4" >> => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => >> "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" => >> "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" => >> [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => "B2_Q2_C1"}], >> "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} >> >> The moment I make it slightly longer, by like appending an extra key "a" >> => "b" it will give me the above error. >> >> It seems to be that the size of the keys plays an issue here, because if >> I convert all the above "choice_id" keys to a mere "c", then even with the >> extra entry "a" => "b" included like following below will work just fine: >> %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => >> "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" => >> "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => >> [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" => >> "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => >> [%{"c" => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => >> [%{"c" => "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => >> [%{"c" => "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], >> "B5_B2_Q3" => [%{"c" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => >> "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => >> [%{"c" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], >> "B5_B2_Q2" => [%{"c" => "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], >> "B2_Q10" => [%{"c" => "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, >> %{"c" => "B4_Q1_C10"}], "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => >> [%{"c" => "B2_Q6_C1"}], "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => >> [%{"c" => "B2_Q1_C1"}], "a" => "b"} >> >> The following on the other hand will just fail: >> data = %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" => "q", >> "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" => "q", >> "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" >> => "q"} >> :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) >> >> The only issue that I was able to track down here is a conjunction of two >> things: >> 1) A limit to the size of total key length inside a map/hash >> 2) sending that hash as a binary over tcp and converting back to term >> >> In all of the above cases, a direct conversion of :erlang.term_to_binary >> followed by :erlang.binary_to_term works just fine without any issues. The >> issue only comes about after that binary was send over tcp and then >> :erlang.binary_to_term is attempted. >> >> I would greatly appreciate if anyone can provide me guidance in this >> matter, point out to me any relevant documentation that officially states >> any of these constraints if they are in fact constraints, and if they are >> not supposed to be constraints then kindly instruct me what additional >> information may be needed for me to get to the bottom of this. >> >> Thank you. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Sun Jan 29 11:50:45 2017 From: erlang@REDACTED (Joe Armstrong) Date: Sun, 29 Jan 2017 11:50:45 +0100 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term In-Reply-To: References: Message-ID: No idea what's wrong - but you could start checking that 1) binary_to_term works locally (it should) 2) the data is transmitted correctly print the md5 checksum of the binary before sending and after reception erlang has a BIF for this erlang:md5 /Joe On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson wrote: > Hi, > > I am using Elixir to send data using :gen_tcp. The syntax presented here may > be elementary Elixir syntax, however the issue is specifically related to > the usage of term_to_binary and binary_to_term conversions over gen_tcp. > > This is the command I used on the client side: >> >> :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > > This is the command I use on the server side: >> >> defp handle_data(data) do >> data >> |> :erlang.binary_to_term >> end > > > When my data gets received on the server side, it seems that at a certain > size threshold that it errors out like this: > >> pry(6)> [error] GenServer #PID<0.502.0> terminating >> >> ** (ArgumentError) argument error >> >> :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, 49, >> 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101, >> 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, ...>>) >> >> (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 >> >> (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 >> >> (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 >> >> Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0, >> 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, >> 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, ...>>} > > > Here is some example data that will just pass fine: > %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"choice_id" > => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => "B2_Q7_C1"}], "B5_B1_Q8" => > [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"choice_id" => > "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => "B5_B2_Q5_C1"}, > %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"choice_id" => > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => "B3_B4_Q1_C1"}], "B2_Q16" > => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => [%{"choice_id" => > "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], "B2_Q9" => > [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => "B2_Q17_C1"}], > "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => > [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => > "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], "B2_Q4" > => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => > "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" => > "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" => > [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => "B2_Q2_C1"}], > "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} > > The moment I make it slightly longer, by like appending an extra key "a" => > "b" it will give me the above error. > > It seems to be that the size of the keys plays an issue here, because if I > convert all the above "choice_id" keys to a mere "c", then even with the > extra entry "a" => "b" included like following below will work just fine: > %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => > "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" => > "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => > [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" => > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => [%{"c" > => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => [%{"c" => > "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => [%{"c" => > "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => [%{"c" > => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => "B3_B3_Q1_C1"}], "B5_B1_Q1" => > [%{"text" => "TEXT ANSWER"}], "B2_Q14" => [%{"c" => "B2_Q14_C1"}], > "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"c" => > "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], "B2_Q10" => [%{"c" => > "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, %{"c" => "B4_Q1_C10"}], > "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => [%{"c" => "B2_Q6_C1"}], > "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => [%{"c" => "B2_Q1_C1"}], "a" => > "b"} > > The following on the other hand will just fail: > data = > %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" > => "q", > "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" > => "q", > "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" > => "q"} > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > The only issue that I was able to track down here is a conjunction of two > things: > 1) A limit to the size of total key length inside a map/hash > 2) sending that hash as a binary over tcp and converting back to term > > In all of the above cases, a direct conversion of :erlang.term_to_binary > followed by :erlang.binary_to_term works just fine without any issues. The > issue only comes about after that binary was send over tcp and then > :erlang.binary_to_term is attempted. > > I would greatly appreciate if anyone can provide me guidance in this matter, > point out to me any relevant documentation that officially states any of > these constraints if they are in fact constraints, and if they are not > supposed to be constraints then kindly instruct me what additional > information may be needed for me to get to the bottom of this. > > Thank you. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From ali.sabil@REDACTED Sun Jan 29 12:56:04 2017 From: ali.sabil@REDACTED (Ali Sabil) Date: Sun, 29 Jan 2017 11:56:04 +0000 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term In-Reply-To: References: Message-ID: You don't seem to be using any framing over TCP. TCP is stream based not message based. Basically what happens is that your message when it gets too large it gets split and sent in multiple packets. You need to add framing around your data to ensure that the entire message is received before you attempt a call to binary_to_term. On Sun, 29 Jan 2017 at 11:50, Joe Armstrong wrote: > No idea what's wrong - but you could start checking that > > 1) binary_to_term works locally (it should) > 2) the data is transmitted correctly > print the md5 checksum of the binary before sending and after reception > erlang has a BIF for this erlang:md5 > > /Joe > > > On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson > wrote: > > Hi, > > > > I am using Elixir to send data using :gen_tcp. The syntax presented here > may > > be elementary Elixir syntax, however the issue is specifically related to > > the usage of term_to_binary and binary_to_term conversions over gen_tcp. > > > > This is the command I used on the client side: > >> > >> :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > > > > > This is the command I use on the server side: > >> > >> defp handle_data(data) do > >> data > >> |> :erlang.binary_to_term > >> end > > > > > > When my data gets received on the server side, it seems that at a certain > > size threshold that it errors out like this: > > > >> pry(6)> [error] GenServer #PID<0.502.0> terminating > >> > >> ** (ArgumentError) argument error > >> > >> :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, > 49, > >> 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101, > >> 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, > ...>>) > >> > >> (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 > >> > >> (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 > >> > >> (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 > >> > >> Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0, > >> 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, > 0, 4, > >> 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, > ...>>} > > > > > > Here is some example data that will just pass fine: > > %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => > [%{"choice_id" > > => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => "B2_Q7_C1"}], > "B5_B1_Q8" => > > [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"choice_id" => > > "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => "B5_B2_Q5_C1"}, > > %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"choice_id" => > > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => "B3_B4_Q1_C1"}], > "B2_Q16" > > => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => [%{"choice_id" => > > "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], "B2_Q9" => > > [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => > "B2_Q17_C1"}], > > "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => > > [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => > > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > > [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => > > "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], > "B2_Q4" > > => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => > > "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" => > > "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" > => > > [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => > "B2_Q2_C1"}], > > "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} > > > > The moment I make it slightly longer, by like appending an extra key "a" > => > > "b" it will give me the above error. > > > > It seems to be that the size of the keys plays an issue here, because if > I > > convert all the above "choice_id" keys to a mere "c", then even with the > > extra entry "a" => "b" included like following below will work just fine: > > %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => > > "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" > => > > "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => > > [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" > => > > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => > [%{"c" > > => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => [%{"c" > => > > "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => [%{"c" => > > "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => > [%{"c" > > => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => "B3_B3_Q1_C1"}], "B5_B1_Q1" > => > > [%{"text" => "TEXT ANSWER"}], "B2_Q14" => [%{"c" => "B2_Q14_C1"}], > > "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"c" => > > "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], "B2_Q10" => [%{"c" => > > "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, %{"c" => "B4_Q1_C10"}], > > "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => [%{"c" => > "B2_Q6_C1"}], > > "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => [%{"c" => "B2_Q1_C1"}], > "a" => > > "b"} > > > > The following on the other hand will just fail: > > data = > > > %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" > > => "q", > > > "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" > > => "q", > > > "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" > > => "q"} > > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > > > The only issue that I was able to track down here is a conjunction of two > > things: > > 1) A limit to the size of total key length inside a map/hash > > 2) sending that hash as a binary over tcp and converting back to term > > > > In all of the above cases, a direct conversion of :erlang.term_to_binary > > followed by :erlang.binary_to_term works just fine without any issues. > The > > issue only comes about after that binary was send over tcp and then > > :erlang.binary_to_term is attempted. > > > > I would greatly appreciate if anyone can provide me guidance in this > matter, > > point out to me any relevant documentation that officially states any of > > these constraints if they are in fact constraints, and if they are not > > supposed to be constraints then kindly instruct me what additional > > information may be needed for me to get to the bottom of this. > > > > Thank you. > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sun Jan 29 16:05:33 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 29 Jan 2017 15:05:33 +0000 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term In-Reply-To: References: Message-ID: My bet is on a framing problem. Joe indirectly gives you a way to handle the question: Test the binary_to_term locally, without the socket. Or send something you know over the wire and check it isn't garbled. This eliminates eventual zlib trouble. The framing problem is subtle: The kernel could either deliver 1 byte a time from the stream, which is inefficient, or it could wait for eternity and just buffer everything never delivering, which is wrong. In practice, the kernel chooses some middle ground. It will deliver bytes from the stream to you either because it has a full buffer internally, or because some time has passed since the last delivery, or that your application is usually going to ask for more data rather quickly. The MTU of your IP datagrams can quickly come into play and you will get messages roughly of MTU size or a multiple hereof. The internet can do evil things to your packets however: fragment them, reorder them, omit a packet in the TCP sequence which requires a SACK roundtrip and so on. These events affects the way the kernel sends bytes to the application in the stream. Since it is a stream, you need your code to handle this case (untested): read_packet(Pkt, Buffer) -> read_buffer(<>). read_buffer(<>) -> {packet, Payload, RestBuffer}; read_buffer(Buffer) -> {need_more_data, Buffer}. write_term(Sock, Term) -> Payload = term_to_term(Term), L = byte_size(Payload), gen_tcp:write(Sock, [<>, Payload]). Coincidentally, this is roughly what the option {packet, 4} does at the VM layer and it is quicker. In the case where we lack enough bytes in the buffer from the kernel, we can tell the world we need more data. If we can break off a packet from the buffer, we do that and return what is remaining in the buffer. The above API requires the caller to repeatedly call read_buffer/1 on the buffer until we can't squeeze more packets out of the buffer. Another variant would just return a list of decoded packets and a buffer state for the things which weren't able to be decoded. On Sun, Jan 29, 2017 at 12:56 PM Ali Sabil wrote: > You don't seem to be using any framing over TCP. TCP is stream based not > message based. > > Basically what happens is that your message when it gets too large it gets > split and sent in multiple packets. You need to add framing around your > data to ensure that the entire message is received before you attempt a > call to binary_to_term. > On Sun, 29 Jan 2017 at 11:50, Joe Armstrong wrote: > > No idea what's wrong - but you could start checking that > > 1) binary_to_term works locally (it should) > 2) the data is transmitted correctly > print the md5 checksum of the binary before sending and after reception > erlang has a BIF for this erlang:md5 > > /Joe > > > On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson > wrote: > > Hi, > > > > I am using Elixir to send data using :gen_tcp. The syntax presented here > may > > be elementary Elixir syntax, however the issue is specifically related to > > the usage of term_to_binary and binary_to_term conversions over gen_tcp. > > > > This is the command I used on the client side: > >> > >> :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > > > > > This is the command I use on the server side: > >> > >> defp handle_data(data) do > >> data > >> |> :erlang.binary_to_term > >> end > > > > > > When my data gets received on the server side, it seems that at a certain > > size threshold that it errors out like this: > > > >> pry(6)> [error] GenServer #PID<0.502.0> terminating > >> > >> ** (ArgumentError) argument error > >> > >> :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, > 49, > >> 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, 101, > >> 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, > ...>>) > >> > >> (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 > >> > >> (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 > >> > >> (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 > >> > >> Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, 0, > >> 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, > 0, 4, > >> 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, > ...>>} > > > > > > Here is some example data that will just pass fine: > > %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => > [%{"choice_id" > > => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => "B2_Q7_C1"}], > "B5_B1_Q8" => > > [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"choice_id" => > > "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => "B5_B2_Q5_C1"}, > > %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"choice_id" => > > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => "B3_B4_Q1_C1"}], > "B2_Q16" > > => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => [%{"choice_id" => > > "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], "B2_Q9" => > > [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => > "B2_Q17_C1"}], > > "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => > > [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => > > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => > > [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => > > "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], > "B2_Q4" > > => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => > > "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" => > > "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" > => > > [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => > "B2_Q2_C1"}], > > "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} > > > > The moment I make it slightly longer, by like appending an extra key "a" > => > > "b" it will give me the above error. > > > > It seems to be that the size of the keys plays an issue here, because if > I > > convert all the above "choice_id" keys to a mere "c", then even with the > > extra entry "a" => "b" included like following below will work just fine: > > %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => > > "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => [%{"c" > => > > "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => > > [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"c" > => > > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => > [%{"c" > > => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => [%{"c" > => > > "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => [%{"c" => > > "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => > [%{"c" > > => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => "B3_B3_Q1_C1"}], "B5_B1_Q1" > => > > [%{"text" => "TEXT ANSWER"}], "B2_Q14" => [%{"c" => "B2_Q14_C1"}], > > "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"c" => > > "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], "B2_Q10" => [%{"c" => > > "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, %{"c" => "B4_Q1_C10"}], > > "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => [%{"c" => > "B2_Q6_C1"}], > > "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => [%{"c" => "B2_Q1_C1"}], > "a" => > > "b"} > > > > The following on the other hand will just fail: > > data = > > > %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" > > => "q", > > > "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > > bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" > > => "q", > > > "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc > > cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" > > => "q"} > > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) > > > > The only issue that I was able to track down here is a conjunction of two > > things: > > 1) A limit to the size of total key length inside a map/hash > > 2) sending that hash as a binary over tcp and converting back to term > > > > In all of the above cases, a direct conversion of :erlang.term_to_binary > > followed by :erlang.binary_to_term works just fine without any issues. > The > > issue only comes about after that binary was send over tcp and then > > :erlang.binary_to_term is attempted. > > > > I would greatly appreciate if anyone can provide me guidance in this > matter, > > point out to me any relevant documentation that officially states any of > > these constraints if they are in fact constraints, and if they are not > > supposed to be constraints then kindly instruct me what additional > > information may be needed for me to get to the bottom of this. > > > > Thank you. > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.greenwoodgeer@REDACTED Sun Jan 29 18:20:57 2017 From: t.greenwoodgeer@REDACTED (Todd Greenwood-Geer) Date: Sun, 29 Jan 2017 09:20:57 -0800 Subject: [erlang-questions] Climate Change Brainstorming Message-ID: Summary ------------- I'm looking for folks interested in brainstorming about Climate Change and various solutions. Feel free to respond to me directly and I'll start a thread with the interested parties. Background ---------------- This video presentation of the latest IPCC report by University of Washington professor, Dr. Richard Gammon really got my attention: https://youtu.be/3Gcf2rqBtLs https://depts.washington.edu/chem/people/faculty/gammon.html Ideas ------- I'm a big fan of Erlang and fault-tolerant distributed systems. This is a really smart group of people that thinks deeply about the longevity and resiliency of systems. There must be a million ways we can use our skills towards this problem. -Todd -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnson786.kevin@REDACTED Sun Jan 29 20:30:00 2017 From: johnson786.kevin@REDACTED (Kevin Johnson) Date: Sun, 29 Jan 2017 14:30:00 -0500 Subject: [erlang-questions] (ArgumentError) argument error :erlang.binary_to_term In-Reply-To: References: Message-ID: It is a newbie mistake on my part. I didn't realize initially that gen_tcp would give me the data in chunks. The conversions are working locally, as I mentioned in my initial question so that settles the zlib issue. When I checked the size before and after sending the data when converted to binary, I saw that before the size is 4051 and after it is only 1460. Obviously it would bomb because I was trying to convert back incomplete data. Setting packet: 4 option as suggested by D?niel Szoboszlay and Jesper Louis Andersen on both server and client fixed the whole issue for me. Thank you all for your valuable input. On Sun, Jan 29, 2017 at 10:05 AM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > My bet is on a framing problem. Joe indirectly gives you a way to handle > the question: Test the binary_to_term locally, without the socket. Or send > something you know over the wire and check it isn't garbled. This > eliminates eventual zlib trouble. > > The framing problem is subtle: The kernel could either deliver 1 byte a > time from the stream, which is inefficient, or it could wait for eternity > and just buffer everything never delivering, which is wrong. In practice, > the kernel chooses some middle ground. It will deliver bytes from the > stream to you either because it has a full buffer internally, or because > some time has passed since the last delivery, or that your application is > usually going to ask for more data rather quickly. The MTU of your IP > datagrams can quickly come into play and you will get messages roughly of > MTU size or a multiple hereof. The internet can do evil things to your > packets however: fragment them, reorder them, omit a packet in the TCP > sequence which requires a SACK roundtrip and so on. These events affects > the way the kernel sends bytes to the application in the stream. > > Since it is a stream, you need your code to handle this case (untested): > > read_packet(Pkt, Buffer) -> > read_buffer(<>). > > read_buffer(<>) -> > {packet, Payload, RestBuffer}; > read_buffer(Buffer) -> > {need_more_data, Buffer}. > > write_term(Sock, Term) -> > Payload = term_to_term(Term), > L = byte_size(Payload), > gen_tcp:write(Sock, [<>, Payload]). > > Coincidentally, this is roughly what the option {packet, 4} does at the VM > layer and it is quicker. In the case where we lack enough bytes in the > buffer from the kernel, we can tell the world we need more data. If we can > break off a packet from the buffer, we do that and return what is remaining > in the buffer. The above API requires the caller to repeatedly call > read_buffer/1 on the buffer until we can't squeeze more packets out of the > buffer. Another variant would just return a list of decoded packets and a > buffer state for the things which weren't able to be decoded. > > On Sun, Jan 29, 2017 at 12:56 PM Ali Sabil wrote: > >> You don't seem to be using any framing over TCP. TCP is stream based not >> message based. >> >> Basically what happens is that your message when it gets too large it >> gets split and sent in multiple packets. You need to add framing around >> your data to ensure that the entire message is received before you attempt >> a call to binary_to_term. >> On Sun, 29 Jan 2017 at 11:50, Joe Armstrong wrote: >> >> No idea what's wrong - but you could start checking that >> >> 1) binary_to_term works locally (it should) >> 2) the data is transmitted correctly >> print the md5 checksum of the binary before sending and after >> reception >> erlang has a BIF for this erlang:md5 >> >> /Joe >> >> >> On Sat, Jan 28, 2017 at 1:55 AM, Kevin Johnson >> wrote: >> > Hi, >> > >> > I am using Elixir to send data using :gen_tcp. The syntax presented >> here may >> > be elementary Elixir syntax, however the issue is specifically related >> to >> > the usage of term_to_binary and binary_to_term conversions over gen_tcp. >> > >> > This is the command I used on the client side: >> >> >> >> :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) >> > >> > >> > This is the command I use on the server side: >> >> >> >> defp handle_data(data) do >> >> data >> >> |> :erlang.binary_to_term >> >> end >> > >> > >> > When my data gets received on the server side, it seems that at a >> certain >> > size threshold that it errors out like this: >> > >> >> pry(6)> [error] GenServer #PID<0.502.0> terminating >> >> >> >> ** (ArgumentError) argument error >> >> >> >> :erlang.binary_to_term(<<131, 116, 0, 0, 0, 28, 109, 0, 0, 0, 5, 66, >> 49, >> >> 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, 0, 4, 116, >> 101, >> >> 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, 87, 69, >> ...>>) >> >> >> >> (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4 >> >> >> >> (stdlib) gen_server.erl:667: :gen_server.handle_msg/5 >> >> >> >> (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 >> >> >> >> Last message: {:tcp, #Port<0.10115>, <<131, 116, 0, 0, 0, 28, 109, 0, >> 0, >> >> 0, 5, 66, 49, 95, 81, 49, 108, 0, 0, 0, 1, 116, 0, 0, 0, 1, 109, 0, 0, >> 0, 4, >> >> 116, 101, 120, 116, 109, 0, 0, 0, 11, 84, 69, 88, 84, 32, 65, 78, 83, >> ...>>} >> > >> > >> > Here is some example data that will just pass fine: >> > %{"B2_Q18" => [%{"choice_id" => "B2_Q18_C1"}], "B3_B9_Q1" => >> [%{"choice_id" >> > => "B3_B9_Q1_C1"}], "B2_Q7" => [%{"choice_id" => "B2_Q7_C1"}], >> "B5_B1_Q8" => >> > [%{"choice_id" => "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"choice_id" => >> > "B3_B6_Q1_C1"}], "B5_B2_Q5" => [%{"choice_id" => "B5_B2_Q5_C1"}, >> > %{"choice_id" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => [%{"choice_id" => >> > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"choice_id" => "B3_B4_Q1_C1"}], >> "B2_Q16" >> > => [%{"choice_id" => "B2_Q16_C1"}], "B2_Q20" => [%{"choice_id" => >> > "B2_Q20_C1"}], "B2_Q5" => [%{"choice_id" => "B2_Q5_C1"}], "B2_Q9" => >> > [%{"choice_id" => "B2_Q9_C1"}], "B2_Q17" => [%{"choice_id" => >> "B2_Q17_C1"}], >> > "B3_B2_Q1" => [%{"choice_id" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => >> > [%{"choice_id" => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"choice_id" => >> > "B3_B3_Q1_C1"}], "B5_B1_Q1" => [%{"text" => "TEXT ANSWER"}], "B2_Q14" => >> > [%{"choice_id" => "B2_Q14_C1"}], "B5_B1_Q3" => [%{"choice_id" => >> > "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"choice_id" => "B5_B2_Q2_C1"}], >> "B2_Q4" >> > => [%{"choice_id" => "B2_Q4_C1"}], "B2_Q10" => [%{"choice_id" => >> > "B2_Q10_C1"}], "B4_Q1" => [%{"choice_id" => "B4_Q1_C1"}, %{"choice_id" >> => >> > "B4_Q1_C10"}], "B5_B1_Q6" => [%{"choice_id" => "B5_B1_Q6_C1"}], "B2_Q6" >> => >> > [%{"choice_id" => "B2_Q6_C1"}], "B2_Q2" => [%{"choice_id" => >> "B2_Q2_C1"}], >> > "B2_Q1" => [%{"choice_id" => "B2_Q1_C1"}]} >> > >> > The moment I make it slightly longer, by like appending an extra key >> "a" => >> > "b" it will give me the above error. >> > >> > It seems to be that the size of the keys plays an issue here, because >> if I >> > convert all the above "choice_id" keys to a mere "c", then even with the >> > extra entry "a" => "b" included like following below will work just >> fine: >> > %{"B2_Q18" => [%{"c" => "B2_Q18_C1"}], "B3_B9_Q1" => [%{"c" => >> > "B3_B9_Q1_C1"}], "B2_Q7" => [%{"c" => "B2_Q7_C1"}], "B5_B1_Q8" => >> [%{"c" => >> > "B5_B1_Q8_C1"}], "B3_B6_Q1" => [%{"c" => "B3_B6_Q1_C1"}], "B5_B2_Q5" => >> > [%{"c" => "B5_B2_Q5_C1"}, %{"c" => "B5_B2_Q5_C2"}], "B3_B7_Q1" => >> [%{"c" => >> > "B3_B7_Q1_C1"}], "B3_B4_Q1" => [%{"c" => "B3_B4_Q1_C1"}], "B2_Q16" => >> [%{"c" >> > => "B2_Q16_C1"}], "B2_Q20" => [%{"c" => "B2_Q20_C1"}], "B2_Q5" => >> [%{"c" => >> > "B2_Q5_C1"}], "B2_Q9" => [%{"c" => "B2_Q9_C1"}], "B2_Q17" => [%{"c" => >> > "B2_Q17_C1"}], "B3_B2_Q1" => [%{"c" => "B3_B2_Q1_C1"}], "B5_B2_Q3" => >> [%{"c" >> > => "B5_B2_Q3_C1"}], "B3_B3_Q1" => [%{"c" => "B3_B3_Q1_C1"}], "B5_B1_Q1" >> => >> > [%{"text" => "TEXT ANSWER"}], "B2_Q14" => [%{"c" => "B2_Q14_C1"}], >> > "B5_B1_Q3" => [%{"c" => "B5_B1_Q3_C22"}], "B5_B2_Q2" => [%{"c" => >> > "B5_B2_Q2_C1"}], "B2_Q4" => [%{"c" => "B2_Q4_C1"}], "B2_Q10" => [%{"c" >> => >> > "B2_Q10_C1"}], "B4_Q1" => [%{"c" => "B4_Q1_C1"}, %{"c" => "B4_Q1_C10"}], >> > "B5_B1_Q6" => [%{"c" => "B5_B1_Q6_C1"}], "B2_Q6" => [%{"c" => >> "B2_Q6_C1"}], >> > "B2_Q2" => [%{"c" => "B2_Q2_C1"}], "B2_Q1" => [%{"c" => "B2_Q1_C1"}], >> "a" => >> > "b"} >> > >> > The following on the other hand will just fail: >> > data = >> > %{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> aaaaaaaaaaaaaaaaaaaaaaaaa" >> > => "q", >> > "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" >> > => "q", >> > "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> ccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc >> cccccccccccccccccccccccccccccccccccccccccccccccc" >> > => "q"} >> > :gen_tcp.send(client_socket, :erlang.term_to_binary(data)) >> > >> > The only issue that I was able to track down here is a conjunction of >> two >> > things: >> > 1) A limit to the size of total key length inside a map/hash >> > 2) sending that hash as a binary over tcp and converting back to term >> > >> > In all of the above cases, a direct conversion of :erlang.term_to_binary >> > followed by :erlang.binary_to_term works just fine without any issues. >> The >> > issue only comes about after that binary was send over tcp and then >> > :erlang.binary_to_term is attempted. >> > >> > I would greatly appreciate if anyone can provide me guidance in this >> matter, >> > point out to me any relevant documentation that officially states any of >> > these constraints if they are in fact constraints, and if they are not >> > supposed to be constraints then kindly instruct me what additional >> > information may be needed for me to get to the bottom of this. >> > >> > Thank you. >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sun Jan 29 23:06:31 2017 From: rvirding@REDACTED (Robert Virding) Date: Sun, 29 Jan 2017 23:06:31 +0100 Subject: [erlang-questions] Inconsistent shadowing of variables in a fun Message-ID: When defining a fun the shadowing of variables occurring in arguments is inconsistent. So with a(X, N) -> fun (<>) -> {X,Y} end. the X in the fun arguments shadows the X from before it while the N is imported and used. An example: 1> c(bt). bt.erl:8: Warning: variable 'X' is unused bt.erl:9: Warning: variable 'X' shadowed in 'fun' {ok,bt} 2> f(G),G=bt:a(34,8). #Fun 3> G(<<1,2,3,4,5>>). {1,2} 4> f(G),G=bt:a(34,16). #Fun 5> G(<<1,2,3,4,5>>). {1,515} Why the difference? Shouldn't we be consistent in shadowing all variables? Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Sun Jan 29 23:42:37 2017 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Sun, 29 Jan 2017 22:42:37 +0000 Subject: [erlang-questions] Inconsistent shadowing of variables in a fun In-Reply-To: References: Message-ID: >From my understanding, the N in this case can never be an unbound variable. If you rename N to M, then it won't compile, so it would not be possible for it to shadow anything. It is similar to map keys. They are always treated as values, even when inside patterns. On Sun, Jan 29, 2017 at 23:07 Robert Virding wrote: > When defining a fun the shadowing of variables occurring in arguments is > inconsistent. So with > > a(X, N) -> > fun (<>) -> {X,Y} end. > > the X in the fun arguments shadows the X from before it while the N is > imported and used. An example: > > 1> c(bt). > bt.erl:8: Warning: variable 'X' is unused > bt.erl:9: Warning: variable 'X' shadowed in 'fun' > {ok,bt} > 2> f(G),G=bt:a(34,8). > #Fun > 3> G(<<1,2,3,4,5>>). > {1,2} > 4> f(G),G=bt:a(34,16). > #Fun > 5> G(<<1,2,3,4,5>>). > {1,515} > > Why the difference? Shouldn't we be consistent in shadowing all variables? > > Robert > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.boaventura@REDACTED Mon Jan 30 00:00:14 2017 From: michel.boaventura@REDACTED (Michel Boaventura) Date: Sun, 29 Jan 2017 21:00:14 -0200 Subject: [erlang-questions] Inconsistent shadowing of variables in a fun In-Reply-To: References: Message-ID: According to Joe's Programming Erlang(page 103): Size must be an expression that evaluates to an integer. In pattern matching, Size must be an integer or a bound variable whose value is an integer. So the N on the pattern match is not a new variable like X. Em 29 de jan de 2017 20:42, "Jos? Valim" escreveu: >From my understanding, the N in this case can never be an unbound variable. If you rename N to M, then it won't compile, so it would not be possible for it to shadow anything. It is similar to map keys. They are always treated as values, even when inside patterns. On Sun, Jan 29, 2017 at 23:07 Robert Virding wrote: > When defining a fun the shadowing of variables occurring in arguments is > inconsistent. So with > > a(X, N) -> > fun (<>) -> {X,Y} end. > > the X in the fun arguments shadows the X from before it while the N is > imported and used. An example: > > 1> c(bt). > bt.erl:8: Warning: variable 'X' is unused > bt.erl:9: Warning: variable 'X' shadowed in 'fun' > {ok,bt} > 2> f(G),G=bt:a(34,8). > #Fun > 3> G(<<1,2,3,4,5>>). > {1,2} > 4> f(G),G=bt:a(34,16). > #Fun > 5> G(<<1,2,3,4,5>>). > {1,515} > > Why the difference? Shouldn't we be consistent in shadowing all variables? > > Robert > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Jan 30 00:07:50 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 30 Jan 2017 12:07:50 +1300 Subject: [erlang-questions] Message send guarantees In-Reply-To: <36c826c2-0c84-4370-9896-4299cb09d5ad@artplant.no> References: <20170126071421.GA46222@erix.ericsson.se> <36c826c2-0c84-4370-9896-4299cb09d5ad@artplant.no> Message-ID: On 27/01/17 11:46 PM, Andrey Tsirulev wrote: > What about the following example? > > A -- msg1 ---> A > A -- msg2 ---> B > B -- msg2 ---> A > > In this case, is it guaranteed that if both msg2 and msg1 get to A's msg > queue, msg1 gets there earlier? Let's start by changing the destination from A to C. A -> msg1 -> C A -> msg2 -> B B -> f(msg2) -> C In this case, it is clearly NOT guaranteed that msg1 will arrive before f(msg2). Why would changing C to A make a difference? Answer: because A -> A is happening in the same place. Let E1 and E2 be two events. E1 happens-before E2 if and only if (1) E1 and E2 happen in the same process E1 happens before E2 in that process's world-line or (2) E1 is the transmission of a message and E2 is the receipt of the same message or (3) There is an event Em such that E1 happens-before Em and Em happens-before E2 If we change the example to A -> msg2 -> B A -> msg1 -> A B -> f(msg2) -> A the transmission of msg1 now happens after the transmission of msg2, so there's no reason to expect it to arrive earlier than f(msg2). Imagine a single-CPU implementation where A -> msg2 -> B puts A to sleep and wakes B, then B -> f(msg2) -> A puts B to sleep and wakes A, then A -> msg1 -> A puts msg1 in A's queue after f(msg2). From ok@REDACTED Mon Jan 30 00:17:56 2017 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 30 Jan 2017 12:17:56 +1300 Subject: [erlang-questions] Inconsistent shadowing of variables in a fun In-Reply-To: References: Message-ID: <8cfbc825-0778-8a36-68ba-e3bdb3212b74@cs.otago.ac.nz> On 30/01/17 11:06 AM, Robert Virding wrote: > When defining a fun the shadowing of variables occurring in arguments is > inconsistent. > > Why the difference? Shouldn't we be consistent in shadowing all variables? I have always wished that Erlang did no shadowing at all. a(X) -> case X+1 of X -> ooh ; _ -> ahh end. b(X) -> (fun (X) -> ooh ; (_) -> ahh end)(X+1). 2> foo:a(0). ahh 3> foo:b(0). ooh Call *that* consistent? I don't. 1> c(foo). foo.erl:8: Warning: this clause cannot match because a previous clause at line 8 always matches foo.erl:8: Warning: variable 'X' is unused foo.erl:8: Warning: variable 'X' shadowed in 'fun' That warning about 'X' being shadowed? LOVE IT LOVE IT LOVE IT! From rvirding@REDACTED Mon Jan 30 01:15:30 2017 From: rvirding@REDACTED (Robert Virding) Date: Mon, 30 Jan 2017 01:15:30 +0100 Subject: [erlang-questions] Inconsistent shadowing of variables in a fun In-Reply-To: References: Message-ID: Yes, the variable bitsize variable must be bound. BUT it can also be bound if it matched "earlier" from inside the same binary. So b() -> fun (<>) -> {N,X} end. and 1> c("bt"). {ok,bt} 2> F = bt:b(). #Fun 3> F(<<8,1,2,3,4>>). {8,1} 4> F(<<5,1,2,3,4>>). {5,0} We see that N doesn't have to be imported. So variables in fun argument patterns aren't imported except for the binary bitsize, which is inconsistent and a bit hacky. Why that type? Robert On 30 January 2017 at 00:00, Michel Boaventura wrote: > According to Joe's Programming Erlang(page 103): Size must be an > expression that evaluates to an integer. In pattern matching, Size must be > an integer or a bound variable whose value is an integer. > > So the N on the pattern match is not a new variable like X. > > > Em 29 de jan de 2017 20:42, "Jos? Valim" > escreveu: > > From my understanding, the N in this case can never be an unbound > variable. If you rename N to M, then it won't compile, so it would not be > possible for it to shadow anything. > > It is similar to map keys. They are always treated as values, even when > inside patterns. > > > On Sun, Jan 29, 2017 at 23:07 Robert Virding wrote: > >> When defining a fun the shadowing of variables occurring in arguments is >> inconsistent. So with >> >> a(X, N) -> >> fun (<>) -> {X,Y} end. >> >> the X in the fun arguments shadows the X from before it while the N is >> imported and used. An example: >> >> 1> c(bt). >> bt.erl:8: Warning: variable 'X' is unused >> bt.erl:9: Warning: variable 'X' shadowed in 'fun' >> {ok,bt} >> 2> f(G),G=bt:a(34,8). >> #Fun >> 3> G(<<1,2,3,4,5>>). >> {1,2} >> 4> f(G),G=bt:a(34,16). >> #Fun >> 5> G(<<1,2,3,4,5>>). >> {1,515} >> >> Why the difference? Shouldn't we be consistent in shadowing all variables? >> >> Robert >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- > > > *Jos? Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Director of R&D > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Jan 30 01:17:32 2017 From: rvirding@REDACTED (Robert Virding) Date: Mon, 30 Jan 2017 01:17:32 +0100 Subject: [erlang-questions] Inconsistent shadowing of variables in a fun In-Reply-To: <8cfbc825-0778-8a36-68ba-e3bdb3212b74@cs.otago.ac.nz> References: <8cfbc825-0778-8a36-68ba-e3bdb3212b74@cs.otago.ac.nz> Message-ID: The shadowing warning is there because it is only there and list comprehensions where variable in patterns are shadowed. No, it isn't consistent but it follows how other functional languages do it, IIRC. The problems with an evolving language. Robert On 30 January 2017 at 00:17, Richard A. O'Keefe wrote: > > > On 30/01/17 11:06 AM, Robert Virding wrote: > >> When defining a fun the shadowing of variables occurring in arguments is >> inconsistent. >> >> Why the difference? Shouldn't we be consistent in shadowing all variables? >> > > I have always wished that Erlang did no shadowing at all. > > a(X) -> > case X+1 of X -> ooh ; _ -> ahh end. > > b(X) -> > (fun (X) -> ooh ; (_) -> ahh end)(X+1). > > > 2> foo:a(0). > ahh > 3> foo:b(0). > ooh > > Call *that* consistent? I don't. > > > 1> c(foo). > foo.erl:8: Warning: this clause cannot match because a previous clause at > line 8 always matches > foo.erl:8: Warning: variable 'X' is unused > foo.erl:8: Warning: variable 'X' shadowed in 'fun' > > That warning about 'X' being shadowed? > > LOVE IT LOVE IT LOVE IT! > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.boaventura@REDACTED Mon Jan 30 01:31:29 2017 From: michel.boaventura@REDACTED (Michel Boaventura) Date: Sun, 29 Jan 2017 22:31:29 -0200 Subject: [erlang-questions] Inconsistent shadowing of variables in a fun In-Reply-To: References: Message-ID: I understand what you are saying. Probably this is allowed on bit matching so you can use parts of the pattern to help match itself. It makes sense if a piece's length depends on other piece value. Em 29 de jan de 2017 22:15, "Robert Virding" escreveu: > Yes, the variable bitsize variable must be bound. BUT it can also be bound > if it matched "earlier" from inside the same binary. So > > b() -> fun (<>) -> {N,X} end. > > and > > 1> c("bt"). > {ok,bt} > 2> F = bt:b(). > #Fun > 3> F(<<8,1,2,3,4>>). > {8,1} > 4> F(<<5,1,2,3,4>>). > {5,0} > > We see that N doesn't have to be imported. So variables in fun argument > patterns aren't imported except for the binary bitsize, which is > inconsistent and a bit hacky. Why that type? > > Robert > > > On 30 January 2017 at 00:00, Michel Boaventura < > michel.boaventura@REDACTED> wrote: > >> According to Joe's Programming Erlang(page 103): Size must be an >> expression that evaluates to an integer. In pattern matching, Size must be >> an integer or a bound variable whose value is an integer. >> >> So the N on the pattern match is not a new variable like X. >> >> >> Em 29 de jan de 2017 20:42, "Jos? Valim" >> escreveu: >> >> From my understanding, the N in this case can never be an unbound >> variable. If you rename N to M, then it won't compile, so it would not be >> possible for it to shadow anything. >> >> It is similar to map keys. They are always treated as values, even when >> inside patterns. >> >> >> On Sun, Jan 29, 2017 at 23:07 Robert Virding wrote: >> >>> When defining a fun the shadowing of variables occurring in arguments is >>> inconsistent. So with >>> >>> a(X, N) -> >>> fun (<>) -> {X,Y} end. >>> >>> the X in the fun arguments shadows the X from before it while the N is >>> imported and used. An example: >>> >>> 1> c(bt). >>> bt.erl:8: Warning: variable 'X' is unused >>> bt.erl:9: Warning: variable 'X' shadowed in 'fun' >>> {ok,bt} >>> 2> f(G),G=bt:a(34,8). >>> #Fun >>> 3> G(<<1,2,3,4,5>>). >>> {1,2} >>> 4> f(G),G=bt:a(34,16). >>> #Fun >>> 5> G(<<1,2,3,4,5>>). >>> {1,515} >>> >>> Why the difference? Shouldn't we be consistent in shadowing all >>> variables? >>> >>> Robert >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> -- >> >> >> *Jos? Valim* >> www.plataformatec.com.br >> Skype: jv.ptec >> Founder and Director of R&D >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunp@REDACTED Mon Jan 30 09:39:19 2017 From: arunp@REDACTED (Arun) Date: Mon, 30 Jan 2017 14:09:19 +0530 Subject: [erlang-questions] Hot code upgrade Message-ID: <588EFBB7.8060903@utl.in> Hi, I am trying to upgrade my application using rebar3 hot code upgrade feature from version 0.0.1 to 0.0.2 and my application consists of around 20 modules, the difference between current version and the new version is that one new module got added and one existing module code has been modified. My doubt is that, in the appup file we have to give list of instructions to upgrade and downgrade software between different versions, is it mandatory that all 20 modules has to be mentioned in the instructions or the modules which are added or modified only need to be mentioned. ? Can anybody assist me on this regards, Arun From raimo+erlang-questions@REDACTED Mon Jan 30 13:26:26 2017 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 30 Jan 2017 13:26:26 +0100 Subject: [erlang-questions] Message send guarantees In-Reply-To: <36c826c2-0c84-4370-9896-4299cb09d5ad@artplant.no> References: <20170126071421.GA46222@erix.ericsson.se> <36c826c2-0c84-4370-9896-4299cb09d5ad@artplant.no> Message-ID: <20170130122626.GA96202@erix.ericsson.se> On Fri, Jan 27, 2017 at 01:46:53PM +0300, Andrey Tsirulev wrote: > Hello, > > 26.01.2017 10:14, Raimo Niskanen wrote: > > On Wed, Jan 25, 2017 at 11:06:05PM +0000, Dmitry Kakurin wrote: > >> No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow. > >> > >> What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why. > > On a multi-threade system it is very hard to know in which order things > > happen, so there is no "reordering" since there is no temporal order > > to begin with. > What about the following example? > > A -- msg1 ---> A > A -- msg2 ---> B > B -- msg2 ---> A > > In this case, is it guaranteed that if both msg2 and msg1 get to A's msg > queue, msg1 gets there earlier? It may be that in the current and all previous implementations msg1 is read before msg2, but it is not _guaranteed_? We had a little discussion and the VM guys after 2 minutes started suggesting possible implementations that could cause msg2 to be read before msg1... http://erlang.org/doc/apps/erts/communication.html 2.1, second paragraph: The only signal ordering guarantee given is the following: if an entity sends multiple signals to the same destination entity, the order is preserved; that is, if A sends a signal S1 to B, and later sends signal S2 to B, S1 is guaranteed not to arrive after S2. > > If this is guaranteed, what makes this example *really* different from > the original example? > > I use this pattern for delayed gen_server's initialization if it's time > consuming, like that: > init(Args) -> > self() ! {init, Args}, > ... > handle_info({init, Args}, State) -> > %% time consuming initialization here > > to return from init ASAP so that start/start_link do not block for a > long time. Then I expect that after > > P = gen_server:start_link(...), > P ! test > > when test is received, gen_server is fully initialized. It never caused > problems, but it might be luck. I'd say that is luck ;-) In this case you might get a message from any client that arrives before your {init,Args} message. This sounds like a use case for proc_lib:spawn..., proc_lib:init_ack/1,2, and gen_server:enter_loop/3-5. Using these you can send an early init_ack to the supervisor, do your time consuming initialization, and then enter the gen_server receive loop when you are ready for the first message. Another way could be to use gen_statem and start in an init state where you postpone all messages until you get your {init,Args} message (there could be a reference() in that message just for fun), do your time consuming initialization and change states to running. Or, of course, design your own event queueing before handling the {init,Args} message. > > Best Regards, > Andrey > Best Regards -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From list1@REDACTED Mon Jan 30 13:53:31 2017 From: list1@REDACTED (Grzegorz Junka) Date: Mon, 30 Jan 2017 12:53:31 +0000 Subject: [erlang-questions] Hot code upgrade In-Reply-To: <588EFBB7.8060903@utl.in> References: <588EFBB7.8060903@utl.in> Message-ID: On 30/01/2017 08:39, Arun wrote: > Hi, > > I am trying to upgrade my application using rebar3 hot code > upgrade feature from version 0.0.1 to 0.0.2 and my application > consists of around 20 modules, the difference between current version > and the new version is that one new module got added and one existing > module code has been modified. My doubt is that, in the appup file we > have to give list of instructions to upgrade and downgrade software > between different versions, is it mandatory that all 20 modules has to > be mentioned in the instructions or the modules which are added or > modified only need to be mentioned. ? > > Can anybody assist me on this Not sure how it works in rebar but usually you don't need to create the relup file yourself. I usually use systools:make_relup. http://erlang.org/doc/man/systools.html You give it two versions of the release files and the location of the applications and it creates the instructions for you. It is smart enough to figure out which modules have been removed/added or which applications have to be stopped/started during the release upgrade. This chapter explains that in more details: http://learnyousomeerlang.com/relups Greg From k.petrauskas@REDACTED Mon Jan 30 13:55:32 2017 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Mon, 30 Jan 2017 14:55:32 +0200 Subject: [erlang-questions] SFTP Message-ID: Hello, I'm trying to use erlang's sftp server and client. In the example bellow, i use erlang for both: the server and the client. For some reason, i can't open some files. 43> ssh_sftp:list_dir(Ch, "."). {ok,["other.txt","some.txt"]} 44> ssh_sftp:read_file(Ch, "some.txt"). {ok,<<"some.txt contents">>} 45> ssh_sftp:read_file(Ch, "other.txt"). {error,no_such_file} I was able to read both files with absolute file names specified, but in my case I should not depend on the absolute paths. I could use the absolute file names when opening files, if I could get the current working directory from the server, but I cant find any function for that. The server behaviour is also a unclear to me. I can't set the cwd, if the root option is used when starting the daemon. I have tried various combinations of root/cwd (cwd is relative to root, or absolute on the target file system), the sftp client always logins with / as its current working directory. In my case, I use the sftpd only for testing the sftp client in CT. I'm working on Erlang/OTP 19 [erts-8.2]. Would be thankful for any pointers. Karolis From ameretat.reith@REDACTED Mon Jan 30 14:02:10 2017 From: ameretat.reith@REDACTED (Ameretat Reith) Date: Mon, 30 Jan 2017 16:32:10 +0330 Subject: [erlang-questions] Hot code upgrade In-Reply-To: <588EFBB7.8060903@utl.in> References: <588EFBB7.8060903@utl.in> Message-ID: On Monday, January 30, 2017 12:09:19 PM IRST, Arun wrote: > ... is it > mandatory that all 20 modules has to be mentioned in the > instructions or the modules which are added or modified only > need to be mentioned. ? Include just changed/removed/added modules. Remember the instruction depends on nature of module, e.g. a library modules often just needs a load but a gen_server most likely needs a synchronous update. From john.hughes@REDACTED Mon Jan 30 16:24:25 2017 From: john.hughes@REDACTED (john.hughes@REDACTED) Date: Mon, 30 Jan 2017 16:24:25 +0100 Subject: [erlang-questions] For your amusement--why doesn't this compile? Message-ID: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> Here?s a behaviour that?s had me tearing my hair today. Consider this module: -module('Aaargh!!'). -define(PLEASE_DONT). -ifndef(PLEASE_DONT). -compile({parse_transform,undefined_parse_transform}). -endif. It fails to compile, with one simple error message: 20> c('Aaargh!!'). Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' error If you can see why, you?re more an Erlang wizard than I am! John -------------- next part -------------- An HTML attachment was scrubbed... URL: From fernando.benavides@REDACTED Mon Jan 30 16:27:24 2017 From: fernando.benavides@REDACTED (Brujo Benavides) Date: Mon, 30 Jan 2017 12:27:24 -0300 Subject: [erlang-questions] For your amusement--why doesn't this compile? In-Reply-To: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> References: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> Message-ID: <962FC93A-9555-4638-8F43-D48DB052D9BA@inakanetworks.com> Well? this on the other hand, compiles: -module(weird). -define(PLEASE_DONT, true). -ifndef(PLEASE_DONT). -compile({parse_transform,undefined_parse_transform}). -endif. > On Jan 30, 2017, at 12:24, wrote: > > Here?s a behaviour that?s had me tearing my hair today. Consider this module: > > -module('Aaargh!!'). > -define(PLEASE_DONT). > -ifndef(PLEASE_DONT). > -compile({parse_transform,undefined_parse_transform}). > -endif. > > It fails to compile, with one simple error message: > > 20> c('Aaargh!!'). > Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' > error > > If you can see why, you?re more an Erlang wizard than I am! > > John > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From fernando.benavides@REDACTED Mon Jan 30 16:29:43 2017 From: fernando.benavides@REDACTED (Brujo Benavides) Date: Mon, 30 Jan 2017 12:29:43 -0300 Subject: [erlang-questions] For your amusement--why doesn't this compile? In-Reply-To: <962FC93A-9555-4638-8F43-D48DB052D9BA@inakanetworks.com> References: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> <962FC93A-9555-4638-8F43-D48DB052D9BA@inakanetworks.com> Message-ID: <56B3A011-201E-4CC8-8C26-DF458AF06616@inakanetworks.com> Strangely enough, this actually generates a different warning: -module(weird). -define(PLEASE_DONT). -ifdef(PLEASE_DONT). -compile({parse_transform,undefined_parse_transform}). -endif. 1> c(weird). src/weird.erl:2: badly formed 'define' > On Jan 30, 2017, at 12:27, Brujo Benavides wrote: > > Well? this on the other hand, compiles: > > -module(weird). > -define(PLEASE_DONT, true). > -ifndef(PLEASE_DONT). > -compile({parse_transform,undefined_parse_transform}). > -endif. > >> On Jan 30, 2017, at 12:24, > > wrote: >> >> Here?s a behaviour that?s had me tearing my hair today. Consider this module: >> >> -module('Aaargh!!'). >> -define(PLEASE_DONT). >> -ifndef(PLEASE_DONT). >> -compile({parse_transform,undefined_parse_transform}). >> -endif. >> >> It fails to compile, with one simple error message: >> >> 20> c('Aaargh!!'). >> Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' >> error >> >> If you can see why, you?re more an Erlang wizard than I am! >> >> John >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Mon Jan 30 16:31:24 2017 From: alex0player@REDACTED (Alex S.) Date: Mon, 30 Jan 2017 18:31:24 +0300 Subject: [erlang-questions] For your amusement--why doesn't this compile? In-Reply-To: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> References: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> Message-ID: <2FA6EE5E-B5FD-4477-9D6B-86E6012E8F8B@gmail.com> > 30 ???. 2017 ?., ? 18:24, ???????(?): > > Here?s a behaviour that?s had me tearing my hair today. Consider this module: > > -module('Aaargh!!'). > -define(PLEASE_DONT). > -ifndef(PLEASE_DONT). > -compile({parse_transform,undefined_parse_transform}). > -endif. > > It fails to compile, with one simple error message: > > 20> c('Aaargh!!'). > Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' > error > > If you can see why, you?re more an Erlang wizard than I am! > Easy enough, there?s no one-argument define(). Define it to true if you don?t care about the value. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hughes@REDACTED Mon Jan 30 16:38:57 2017 From: john.hughes@REDACTED (john.hughes@REDACTED) Date: Mon, 30 Jan 2017 16:38:57 +0100 Subject: [erlang-questions] For your amusement--why doesn't this compile? In-Reply-To: <2FA6EE5E-B5FD-4477-9D6B-86E6012E8F8B@gmail.com> References: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> <2FA6EE5E-B5FD-4477-9D6B-86E6012E8F8B@gmail.com> Message-ID: <2006F0B5897642CEAAED368B94F5643B@JohnsTablet2014> So why isn?t that reported as an error? John From: Alex S. Sent: Monday, January 30, 2017 4:31 PM To: john.hughes@REDACTED Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] For your amusement--why doesn't this compile? 30 ???. 2017 ?., ? 18:24, ???????(?): Here?s a behaviour that?s had me tearing my hair today. Consider this module: -module('Aaargh!!'). -define(PLEASE_DONT). -ifndef(PLEASE_DONT). -compile({parse_transform,undefined_parse_transform}). -endif. It fails to compile, with one simple error message: 20> c('Aaargh!!'). Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' error If you can see why, you?re more an Erlang wizard than I am! Easy enough, there?s no one-argument define(). Define it to true if you don?t care about the value. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Mon Jan 30 16:50:48 2017 From: alex0player@REDACTED (Alex S.) Date: Mon, 30 Jan 2017 18:50:48 +0300 Subject: [erlang-questions] For your amusement--why doesn't this compile? In-Reply-To: <2006F0B5897642CEAAED368B94F5643B@JohnsTablet2014> References: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> <2FA6EE5E-B5FD-4477-9D6B-86E6012E8F8B@gmail.com> <2006F0B5897642CEAAED368B94F5643B@JohnsTablet2014> Message-ID: <2D1DC52E-3E39-485D-B02F-ED55FA9EEE2D@gmail.com> It?s considered an unknown module attribute, and those aren?t errors. > 30 ???. 2017 ?., ? 18:38, john.hughes@REDACTED ???????(?): > > So why isn?t that reported as an error? > > John > > From: Alex S. > Sent: Monday, January 30, 2017 4:31 PM > To: john.hughes@REDACTED > Cc: erlang-questions@REDACTED > Subject: Re: [erlang-questions] For your amusement--why doesn't this compile? > > >> 30 ???. 2017 ?., ? 18:24, > > ???????(?): >> >> Here?s a behaviour that?s had me tearing my hair today. Consider this module: >> >> -module('Aaargh!!'). >> -define(PLEASE_DONT). >> -ifndef(PLEASE_DONT). >> -compile({parse_transform,undefined_parse_transform}). >> -endif. >> >> It fails to compile, with one simple error message: >> >> 20> c('Aaargh!!'). >> Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' >> error >> >> If you can see why, you?re more an Erlang wizard than I am! >> > Easy enough, there?s no one-argument define(). > > Define it to true if you don?t care about the value. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hughes@REDACTED Mon Jan 30 16:55:06 2017 From: john.hughes@REDACTED (john.hughes@REDACTED) Date: Mon, 30 Jan 2017 16:55:06 +0100 Subject: [erlang-questions] For your amusement--why doesn't this compile? In-Reply-To: <2D1DC52E-3E39-485D-B02F-ED55FA9EEE2D@gmail.com> References: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> <2FA6EE5E-B5FD-4477-9D6B-86E6012E8F8B@gmail.com> <2006F0B5897642CEAAED368B94F5643B@JohnsTablet2014> <2D1DC52E-3E39-485D-B02F-ED55FA9EEE2D@gmail.com> Message-ID: Actually, it is an error. As you can see by compiling this instead: -module('Aaargh!!'). -define(PLEASE_DONT). -ifndef(PLEASE_DONT). -error("Weird!"). -endif. 2> c('Aaargh!!'). Aaargh!!.erl:2: badly formed 'define' Aaargh!!.erl:4: -error("Weird!"). error Curiouser and curiouser... John From: Alex S. Sent: Monday, January 30, 2017 4:50 PM To: john.hughes@REDACTED Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] For your amusement--why doesn't this compile? It?s considered an unknown module attribute, and those aren?t errors. 30 ???. 2017 ?., ? 18:38, john.hughes@REDACTED ???????(?): So why isn?t that reported as an error? John From: Alex S. Sent: Monday, January 30, 2017 4:31 PM To: john.hughes@REDACTED Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] For your amusement--why doesn't this compile? 30 ???. 2017 ?., ? 18:24, ???????(?): Here?s a behaviour that?s had me tearing my hair today. Consider this module: -module('Aaargh!!'). -define(PLEASE_DONT). -ifndef(PLEASE_DONT). -compile({parse_transform,undefined_parse_transform}). -endif. It fails to compile, with one simple error message: 20> c('Aaargh!!'). Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' error If you can see why, you?re more an Erlang wizard than I am! Easy enough, there?s no one-argument define(). Define it to true if you don?t care about the value. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Mon Jan 30 17:00:21 2017 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Mon, 30 Jan 2017 17:00:21 +0100 Subject: [erlang-questions] For your amusement--why doesn't this compile? In-Reply-To: <2D1DC52E-3E39-485D-B02F-ED55FA9EEE2D@gmail.com> References: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> <2FA6EE5E-B5FD-4477-9D6B-86E6012E8F8B@gmail.com> <2006F0B5897642CEAAED368B94F5643B@JohnsTablet2014> <2D1DC52E-3E39-485D-B02F-ED55FA9EEE2D@gmail.com> Message-ID: My bet is that the parse transform fails before a warning/error is emitted for the bad -define declaration. I am not sure there is much it could do, given the parse_transform itself could rewrite -define(Foo) to mean something else, disappearing with the error in the first place. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Mon, Jan 30, 2017 at 4:50 PM, Alex S. wrote: > It?s considered an unknown module attribute, and those aren?t errors. > > 30 ???. 2017 ?., ? 18:38, john.hughes@REDACTED ???????(?): > > So why isn?t that reported as an error? > > John > > *From:* Alex S. > *Sent:* Monday, January 30, 2017 4:31 PM > *To:* john.hughes@REDACTED > *Cc:* erlang-questions@REDACTED > *Subject:* Re: [erlang-questions] For your amusement--why doesn't this > compile? > > > > 30 ???. 2017 ?., ? 18:24, > ???????(?): > > Here?s a behaviour that?s had me tearing my hair today. Consider this > module: > > -module('Aaargh!!'). > -define(PLEASE_DONT). > -ifndef(PLEASE_DONT). > -compile({parse_transform,undefined_parse_transform}). > -endif. > > It fails to compile, with one simple error message: > > 20> c('Aaargh!!'). > Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' > error > > If you can see why, you?re more an Erlang wizard than I am! > > > Easy enough, there?s no one-argument define(). > > Define it to true if you don?t care about the value. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hughes@REDACTED Mon Jan 30 17:05:21 2017 From: john.hughes@REDACTED (john.hughes@REDACTED) Date: Mon, 30 Jan 2017 17:05:21 +0100 Subject: [erlang-questions] For your amusement--why doesn't this compile? In-Reply-To: References: <49C45765432F4B68A50ABFFC60A60983@JohnsTablet2014> <2FA6EE5E-B5FD-4477-9D6B-86E6012E8F8B@gmail.com> <2006F0B5897642CEAAED368B94F5643B@JohnsTablet2014> <2D1DC52E-3E39-485D-B02F-ED55FA9EEE2D@gmail.com> Message-ID: <68EBB2668AEE49D19FA61921558F537B@JohnsTablet2014> I think you?re right?this is our current hypothesis. I do think it would be kinder to complain about the ?define, in the case where the parse transform fails... since it?s the error in the define which CAUSES the parse transform to fail! Maybe a failing parse_transform could be treated as an identity function as far as other errors are concerned... although, of course, there could be a LOT of other errors caused by the parse transform failing to run! Damned if you do, damned if you don?t! Thanks for joining in the fun John From: Jos? Valim Sent: Monday, January 30, 2017 5:00 PM To: Alex S. Cc: John Hughes ; Erlang Subject: Re: [erlang-questions] For your amusement--why doesn't this compile? My bet is that the parse transform fails before a warning/error is emitted for the bad -define declaration. I am not sure there is much it could do, given the parse_transform itself could rewrite -define(Foo) to mean something else, disappearing with the error in the first place. Jos? Valim www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Mon, Jan 30, 2017 at 4:50 PM, Alex S. wrote: It?s considered an unknown module attribute, and those aren?t errors. 30 ???. 2017 ?., ? 18:38, john.hughes@REDACTED ???????(?): So why isn?t that reported as an error? John From: Alex S. Sent: Monday, January 30, 2017 4:31 PM To: john.hughes@REDACTED Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] For your amusement--why doesn't this compile? 30 ???. 2017 ?., ? 18:24, ???????(?): Here?s a behaviour that?s had me tearing my hair today. Consider this module: -module('Aaargh!!'). -define(PLEASE_DONT). -ifndef(PLEASE_DONT). -compile({parse_transform,undefined_parse_transform}). -endif. It fails to compile, with one simple error message: 20> c('Aaargh!!'). Aaargh!!.erl: undefined parse transform 'undefined_parse_transform' error If you can see why, you?re more an Erlang wizard than I am! Easy enough, there?s no one-argument define(). Define it to true if you don?t care about the value. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: wlEmoticon-smile[1].png Type: image/png Size: 1046 bytes Desc: not available URL: From cean.ebengt@REDACTED Mon Jan 30 17:47:31 2017 From: cean.ebengt@REDACTED (bengt) Date: Mon, 30 Jan 2017 17:47:31 +0100 Subject: [erlang-questions] RFC2630 ASN.1, how to decode? Message-ID: <9A57B66A-46FC-4B82-98C3-172A9979EC09@gmail.com> Greetings, Has anybody decoded RFC2630 and would be inclined to give pointers on how to go about it? Yes, RFC2630 is obsoleted, but the input data use it. erlang:system_info(system_version). "Erlang/OTP 19 [erts-8.2] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false]\n? I create a Data.asn file with ASN contents from the end of https://tools.ietf.org/html/rfc2630 There are 2 IMPORT statements, like this: -- Directory Information Framework (X.501) Name FROM InformationFramework { joint-iso-itu-t ds(5) modules(1) informationFramework(1) 3 } -- Directory Authentication Framework (X.509) AlgorithmIdentifier, AttributeCertificate, Certificate, CertificateList, CertificateSerialNumber FROM AuthenticationFramework { joint-iso-itu-t ds(5) module(1) authenticationFramework(7) 3 } ; I have located 2 links: https://www.itu.int/ITU-T/formal-language/itu-t/x/x501/2012/InformationFramework.html https://www.itu.int/ITU-T/formal-language/itu-t/x/x509/2016/AuthenticationFramework.html Each links content has been copied into an Erlang .asn file: AuthenticationFramework.asn InformationFramework.asn When I compile (erlc Data.asn) I get: Internal error: error:{badrecord,typedef} [{asn1ct_check,check_type,3,[{file,"asn1ct_check.erl"},{line,2544}]}, {asn1ct_check,check_type,3,[{file,"asn1ct_check.erl"},{line,2558}]}, {asn1ct_check,check_setof,3,[{file,"asn1ct_check.erl"},{line,4637}]}, {asn1ct_check,check_type,3,[{file,"asn1ct_check.erl"},{line,2772}]}, {asn1ct_check,check_type,3,[{file,"asn1ct_check.erl"},{line,2558}]}, {asn1ct_check,check_each_component2,4, [{file,"asn1ct_check.erl"},{line,4863}]}, {asn1ct_check,check_sequence,3,[{file,"asn1ct_check.erl"},{line,4360}]}, {asn1ct_check,check_type,3,[{file,"asn1ct_check.erl"},{line,2740}]}] erlc: exit 1 If I remove the IMPORT ... FROM AuthenticationFramework the crash goes away and I get undefined's for the no longer imported items, instead. Is the copy of contents from the two links, to Erlang asn files, too simplistic? The Erlang ASN example says to start with Data DEFINITIONS AUTOMATIC TAGS ::= BEGIN whereas the links start with AuthenticationFramework {joint-iso-itu-t ds(5) module(1) authenticationFramework(7) 8} DEFINITIONS ::= BEGIN -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Jan 30 18:30:52 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 30 Jan 2017 18:30:52 +0100 Subject: [erlang-questions] Common Test logs Message-ID: <5b05e1d9-6cf5-8ad5-5321-22f06bdabe5d@ninenines.eu> Hello, I am looking for a Common Test option that allows me to restrict the number of logs Common Test keeps around. In practice I don't always care about the test runs I did last month, I only want the 10 or so most recents. I would like to configure Common Test so that it deletes older runs automatically. Today I eventually "rm -rf logs" when it gets too big, but it would be good if I didn't have to think about it. Is there such an option (perhaps undocumented) and if not, can it be considered adding it? This way when ct_run starts it can cleanup automatically. Thanks, -- Lo?c Hoguin https://ninenines.eu From cean.ebengt@REDACTED Mon Jan 30 19:01:04 2017 From: cean.ebengt@REDACTED (bengt) Date: Mon, 30 Jan 2017 19:01:04 +0100 Subject: [erlang-questions] SFTP In-Reply-To: References: Message-ID: Greetings, Can you try with sshd, instead of ssh_sftpd ? As a way of halving the potential error sources. Also, which Erlang version are you on? bengt > On 30 Jan 2017, at 13:55, Karolis Petrauskas wrote: > > Hello, > > I'm trying to use erlang's sftp server and client. In the example > bellow, i use erlang for both: the server and the client. For some > reason, i can't open some files. > > 43> ssh_sftp:list_dir(Ch, "."). > {ok,["other.txt","some.txt"]} > > 44> ssh_sftp:read_file(Ch, "some.txt"). > {ok,<<"some.txt contents">>} > > 45> ssh_sftp:read_file(Ch, "other.txt"). > {error,no_such_file} > > I was able to read both files with absolute file names specified, but > in my case I should not depend on the absolute paths. I could use the > absolute file names when opening files, if I could get the current > working directory from the server, but I cant find any function for > that. > > The server behaviour is also a unclear to me. I can't set the cwd, if > the root option is used when starting the daemon. I have tried various > combinations of root/cwd (cwd is relative to root, or absolute on the > target file system), the sftp client always logins with / as its > current working directory. > > In my case, I use the sftpd only for testing the sftp client in CT. > I'm working on Erlang/OTP 19 [erts-8.2]. > > Would be thankful for any pointers. > > Karolis > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From k.petrauskas@REDACTED Mon Jan 30 21:19:03 2017 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Mon, 30 Jan 2017 22:19:03 +0200 Subject: [erlang-questions] SFTP In-Reply-To: References: Message-ID: I'm using "Erlang/OTP 19 [erts-8.2]", debian package from erlang-solutions, version 1:19.2-1. I'll check that with erlang client and sshd server. I was playing with ssh_sftpd server from a command line (sftp), and noticed no problems except the cwd problem when combined with root directory restriction. Karolis On Mon, Jan 30, 2017 at 8:01 PM, bengt wrote: > Greetings, > > Can you try with sshd, instead of ssh_sftpd ? As a way of halving the potential error sources. Also, which Erlang version are you on? > > > bengt > >> On 30 Jan 2017, at 13:55, Karolis Petrauskas wrote: >> >> Hello, >> >> I'm trying to use erlang's sftp server and client. In the example >> bellow, i use erlang for both: the server and the client. For some >> reason, i can't open some files. >> >> 43> ssh_sftp:list_dir(Ch, "."). >> {ok,["other.txt","some.txt"]} >> >> 44> ssh_sftp:read_file(Ch, "some.txt"). >> {ok,<<"some.txt contents">>} >> >> 45> ssh_sftp:read_file(Ch, "other.txt"). >> {error,no_such_file} >> >> I was able to read both files with absolute file names specified, but >> in my case I should not depend on the absolute paths. I could use the >> absolute file names when opening files, if I could get the current >> working directory from the server, but I cant find any function for >> that. >> >> The server behaviour is also a unclear to me. I can't set the cwd, if >> the root option is used when starting the daemon. I have tried various >> combinations of root/cwd (cwd is relative to root, or absolute on the >> target file system), the sftp client always logins with / as its >> current working directory. >> >> In my case, I use the sftpd only for testing the sftp client in CT. >> I'm working on Erlang/OTP 19 [erts-8.2]. >> >> Would be thankful for any pointers. >> >> Karolis >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From dima_kakurin@REDACTED Tue Jan 31 02:43:07 2017 From: dima_kakurin@REDACTED (Dmitry Kakurin) Date: Tue, 31 Jan 2017 01:43:07 +0000 Subject: [erlang-questions] Message send guarantees In-Reply-To: <20170127074132.GA6098@erix.ericsson.se> References: , <20170127074132.GA6098@erix.ericsson.se> Message-ID: On Thu, Jan 26, 2017 at 11:41 PM, Raimo Niskanen > wrote: Have you changed your question? Previously the problem was to send non-proxied messages after proxied messages. I've used inserting proxy in my original email (as it was the smallest setup I can distill my question to), and removing proxy in followup. Sorry about confusion. The problems are actually isomorphic and have the same solution :-). But if B is a message relay (receive Msg -> C ! Msg, loop() end), and provided the ping/pong is a gen_server call (since the request contains who should have the reply, then this should also work: 1. A --msg1---> B 2. A --req----> B 3. B --msg1---> C 4. B --req----> C 5. A <--reply-- C 6. A --msg2---> C I had the same solution in mind. Also it's a good idea and optimization to reply directly from C to A in step 5 instead of going thru B. I didn't think of that, thanks. > Now the only problem I have to solve is: what can be used as a benign message that I can send to any gen_server such that it would reply (to emulate ping/pong exchange above)? sys:get_status/1,2 returns much more data including what sys:get_state/1,2 returns, so it is less suitable. If you know anything more about your gen_server C, it may have some status quering handle_call that can be used. Maybe sys:statistics/2 can be good enough. Mostly no statistics collection is activated for a given gen_server so you will get just {ok,no_statistics} back as pong, and if statistics collection should be activated the gen_server does not much more than process_info(self(), reductions) and erlang:localtime(), which may and may not be regarded as heavy, but compared to returning the whole arbitrary gen_server state it may be better. After digging thru sys module code a little more I've settled on using sys:remove(C,nil). It's asking to remove debug function 'nil' that is of course bogus, but returns ok quickly and is a single list.remove operation from a list that's usually empty. Thank you, Dmitry ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Raimo Niskanen Sent: Thursday, January 26, 2017 11:41 PM To: erlang-questions@REDACTED Subject: Re: [erlang-questions] Message send guarantees On Fri, Jan 27, 2017 at 02:25:49AM +0000, Dmitry Kakurin wrote: > From: Joe Armstrong > > Sent: Thursday, January 26, 2017 5:29 AM > > On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin > wrote: > > When I execute "pid ! msg" and it returns, what are the guarantees if pid > > and self are on the same node? > > Can I assume that by the time it returns, msg is in the pid's queue? And as > > a result all messages sent to pid afterwards will be queued after msg? > > > > The reason I'm asking is because I need a guarantee of proper message > > ordering in the following scenario when all processes involved are local to > > a node: > > 1. A --msg1--> C (process A sends msg1 to process C) > > 2. A --msg2--> B > > 3. B --msg2--> C (process B simply proxies msg2 from A to C) > > > > I need a guarantee that msg2 will always appear in C's queue after msg1. > > This cannot be guaranteed > > Thanks Joe and others. I understand now that it's fruitless to pursue a direction where I'd try to somehow artificially establish this guarantee. > > I have to re-evaluate my constraints then. All along my main constraint was that C can be any gen_server, it's not owned by me and I cannot change it. > If I could change C then the following would work: > 1. A --msg1--> C > 2. A --ping--> C > 3. A <--pong-- C > 4. A --msg2--> B > 5. B --msg2--> C > Have you changed your question? Previously the problem was to send non-proxied messages after proxied messages. But if B is a message relay (receive Msg -> C ! Msg, loop() end), and provided the ping/pong is a gen_server call (since the request contains who should have the reply, then this should also work: 1. A --msg1---> B 2. A --req----> B 3. B --msg1---> C 4. B --req----> C 5. A <--reply-- C 6. A --msg2---> C > All we need from C is to respond "pong" to "ping" message. In this case A knows msg2 is enqueued after msg1 because A has observed "pong" in response to "ping" that was sent after msg1. The ordering is as desired here. > Now the only problem I have to solve is: what can be used as a benign message that I can send to any gen_server such that it would reply (to emulate ping/pong exchange above)? > Looking thru gen_server source code I see that it's processing sys messages. So I can use get_state message as "ping" and response with state as "pong". Except it may not be really benign perf-wise if amount of state kept by C is significant. > There is also get_status, but I don't understand how it's processed and if I can use it for ping/pong purposes. Can I? Is there a better message? sys:get_status/1,2 returns much more data including what sys:get_state/1,2 returns, so it is less suitable. If you know anything more about your gen_server C, it may have some status quering handle_call that can be used. Maybe sys:statistics/2 can be good enough. Mostly no statistics collection is activated for a given gen_server so you will get just {ok,no_statistics} back as pong, and if statistics collection should be activated the gen_server does not much more than process_info(self(), reductions) and erlang:localtime(), which may and may not be regarded as heavy, but compared to returning the whole arbitrary gen_server state it may be better. > > Do you have a better idea for ping/pong messages on C? Or how to solve this in general? > > Thank you, Dmitry. > > P.S. I know that in this simplified example I could piggyback on msg1 if it returns something but in real code I cannot, so let's assume it's a one-way message (a cast, not a call). So I'm OK with ping/pong. > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdl.web@REDACTED Tue Jan 31 05:22:09 2017 From: sdl.web@REDACTED (Leo Liu) Date: Tue, 31 Jan 2017 12:22:09 +0800 Subject: [erlang-questions] Common Test logs References: <5b05e1d9-6cf5-8ad5-5321-22f06bdabe5d@ninenines.eu> Message-ID: On 2017-01-30 18:30 +0100, Lo?c Hoguin wrote: > In practice I don't always care about the test runs I did last month, > I only want the 10 or so most recents. I would like to configure > Common Test so that it deletes older runs automatically. Today I > eventually "rm -rf logs" when it gets too big, but it would be good if > I didn't have to think about it. I have been wanting the same thing a few times. Thanks for putting this forward. Leo From zxq9@REDACTED Tue Jan 31 07:49:30 2017 From: zxq9@REDACTED (zxq9) Date: Tue, 31 Jan 2017 15:49:30 +0900 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: <8dc034b0-1aa4-4b1d-a5f2-a5256bf16f48@Spark> References: <12f0db3b-989e-a8fa-0017-f0213b27e155@cs.otago.ac.nz> <8dc034b0-1aa4-4b1d-a5f2-a5256bf16f48@Spark> Message-ID: <2929868.8ykAR5J0xW@changa> On 2017?1?17? ??? 08:17:10 Micha? Muska?a wrote: > On 17 Jan 2017, 07:39 +0100, Richard A. O'Keefe , wrote: > > > > > Let's see, I use Fortran, C, occasionally C++ or Objective C, > > Python, Prolog, Erlang, Lisp, Scheme, R, Ada (when Apple haven't > > broken my gcc setup *again*), Smalltalk, Java, C#, SML, F#, > > sometimes a bit of JavaScript (yuck). In *all* of them, if > > there are operations to convert case, the operations are in fact > > *library* operations. In *all* of them, I have to look them up. > > > > I just checked and all: Python 3, Ruby, Java, C# (so probably F# as well) and JavaScript properly uppercase the character "?" using the built-in functions, without any additional libraries. Those libraries may upper/lower your name, but none of them have any clue about the equivalence of 32 ??? ??? ?? Not to mention REALLY EASY one-for-one stuff like Nihon NIHON ??? ??? ??? The equivalent of length() doesn't even come out the same over the last three in some langauges. And which of those is "upper case"? And why don't we have a ready conversion for them? And why must ???? be 4 "characters" in hirgana but convert to 5 in halfwidth katakana ????? but still 4 in full-width katakana ????. etc. In almost every different programming language/environment I use I am required to write some conversion utilities that cover the gazillion direct, "simple" conversions that exist for my primary language -- and Japanese users are actually used to expecting less out of life and have simply come to accept that computers cannot provide an interface that approximates the language in use. In a few cases a wrapper to an input method library can help (Anthy and mozc libs, for example, have almost all of this covered already, but only for Japanese) but often it is just about as hard to write wrappers as it is to just write conversions in the language being used (and same-language code is dramatically easier for another programmer to understand and extend later on when something changes). While unicode libs often cover many of the latinish cases they simply ignore any instances where "case" is not a binary concept -- and those are the majority in Asia. EVERYTHING is an extended library here, no language or platform has this stuff built in (except for platforms from Nintendo, because they are magical unicorns who actually pay for enormous amounts of refinement work). So if it isn't going to handle ANY proper text conversions for the half of the world that lives outside the Western(ish) language sphere, why should built-ins be expected to be aware of >127 conversions at all if it is only going to ever cover a tiny fraction of them? Don't misconstrue this as an argument for "fairness" -- whatever that would even mean. I am arguing for a lack of surprise. It is surprising to me that there is this idea of "upper" and "lower" case conversions that are implicit and built-in, but that there is no concept of script casting in general when it is the norm in so many non-European languages. Functions like upper() and lower() are just special cases of script casting and only make sense in the context of Latin/Greek(ish) alphabets. A more general function like scriptcast(String, ScriptName) is actually the real solution that covers all of these. The algorithmic complexity of such a function is close to zero, but the number of cases that must be hand- or rule-written is enormous. I don't understand arguing for a handful of non-trivial ones and then totally ignoring the vast sea of other equally necessary conversions. Blah blah blah... Anything more than an upper()/lower() that covers codepoints <128 should never be included in a standard library. upper()/lower() are necessary for case-insensitive string-based instruction standards like HTTP/1.1 (Whose freaking idea was it to make header labels case-insensitive anyway?!?). Outside of that I just can't see anything other than an external lib that implements a scriptcast/2,3 and comes bundled with a neverending river of conversion clauses. -Craig From zxq9@REDACTED Tue Jan 31 09:52:08 2017 From: zxq9@REDACTED (zxq9) Date: Tue, 31 Jan 2017 17:52:08 +0900 Subject: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead In-Reply-To: References: <2929868.8ykAR5J0xW@changa> Message-ID: <3747678.5iJU0ezvv9@changa> On 2017?1?31? ??? 07:43:05 you wrote: > zxq9/Craig, > > > While unicode libs often cover many of the latinish cases they simply ignore any instances where "case" is not a binary concept -- and those are the majority in Asia. > > Since I only know languages written with latin characters, could you just giving some examples on what conversions you mean and in what circumstances they are needed? > For example I don't see/understand why 32 and the big 32 isn't just a font size issue. :-) > > I fully agree on your frustration regarding protocols with header strings that are allowed to have different casing. Hi, Ola. 32 and ?? are totally different codepoints, not just font size issues. 11> $3. 51 12> $2. 50 13> $?. 65299 14> $?. 65298 So "32" is [51,50] while "??" is [65299,65298]. When accepting user input or reading a page from a book or interpreting a webpage or data stored in a database or a player profile that is arriving as JSON or a chat room name or whatever -- any case where you may need to unambiguously interpret a string as an integer -- you need to know the difference. These are NOT font differences, these are actually different characters (codepoints, really) that have the same value to humans but are not interpreted as the same value by the computer. So which one of these is "upper case"? Neither. There are several different sets of numeric glyphs in unicode that all mean 0-9 to humans. Which one is the "right" one? How do I verify totally valid user input in this environment? Why should programs written to work in Russia accept text that represents numbers there regardless what input mode the user is in, but require that Japanese (among many others) users to only input "half-width" numerals? Not only that, internationalization of any user-facing program will require that the input fields not only say "Enter your Foo Number" in the correct language, somehow the whole interface will be required to display an extra, special message that only occurs when the user's locale is set to a subset of special input locales that happen to have full-width text numerals in addition to the standard ASCII values. See the problem? That is just one problem. The lack of actual script casting VS only the special case of upper() and lower() means that I cannot use any unicode library function to compare two exactly equivalent strings that represent a user's name in sound-spelling. There is no unicode:compare("???", "???"), for example, even though they are directly equivalent, codepoint for codepoint. This is directly analogous to performing a case-insensitive comparison of compare("foo", "FOO") -- but it simply is not possible in Japanese without writing a library for it. There are indirect equivalencies as well, such as compare("???", "????") -- one is a list of three codepoints, the other a list of four codepoints, but they are directly equivalent and ordinary to receive as input from users or databases. In the exact same way, Japanese libraries typically have a romanized indrect comparison for, say, compare("honda", "???"). Kanji comparisons typically only exist in heavy-duty text processing and text search libraries and can only give a probability match or a boolean possible match based on a reading table of possible character soundings -- I would never expect kanji comparisons in a simple unicode library. Also, person and place proper names are notoriously difficult for anything but the human brain to disambiguate (and even then you only know what you already know -- our minds aren't yet equipped with wifi). An issue we haven't touched on yet is that unicode allows characters to be composed or explicit. So the glyph we think of as "?", for example, may be the codepoint "?" (which is the most common to encounter) or the characters "?" and "?" with a special marker that tells the system reading it that they belong together. Once again, they are directly equivalent and legal input from users or non-human data systems and SHOULD be able to compare equivalently, but they are totally different in representation -- just because. And both are legal unicode. This difference is prevalent in Chinese intermediate user input because several stroke (radical) based input systems exist alongside complete glyph systems -- they typically are resolved to a single characters before arriving at your program, but not always. In Korean characters can be composites or explicit as well, and since the entire hangul writing system is based on compositions and the keyboard is basically split down the middle between the vowel-ish ones and the consonant-ish ones, providing partial-input search is predicated on interpretation of partials. In practice we can often control the situation just enough to get at least some useful work done, but that doesn't change the fact that the situation is actually a bit nightmarish and silly and we shouldn't be reinventing this particular wheel all the time in 2017. That users have become accustomed to expecting all data systems to be totally screwed up and stupid is a powerful indictment of the art of text processing today. Japanese users simply accept that this is just how the world is -- at least the world outside of video games and closed platforms. That's also why you don't see a lot of Japanese business software outside Japan, and rarely see any of the popular American business programs inside Japan -- and internationalization of games is hard enough that quite a few of them are intended for the domestic market or the international market, but usually not both at the same time. The mess of data issues is just a big pile of poo and often isn't worth dealing with outside of mega projects. Contemplate that last bit for a while. This is really a stupid reason to constrain the software market, but that's just how things are -- mostly because very few string comparison or conversion libs actually do script casting and instead focus on this one special case of upper() and lower(). (Don't get me started on calendar libraries or alternative numeric representations that have totally unambiguous representations...) Anyway, I understand the situation and have simply had to write libraries to deal with this in several programming environments already -- I have exactly zero hope that anything will get better in the future. The annoying part is doing that over and over because most of the things I write wind up becoming proprietary, so the next time I go somewhere else I write the same stuff again (often there is a prohibition against using FOSS and also I am usually not very excited about writing some MIT-licensed stuff for free on github between projects when I have a gigantic personal backlog already). It should SURPRISE me when a built-in upper() or lower() function in a language operates on more than just ASCII for the purpose of text protocol interpretation. If an upper() or lower() function DOES work on, say, Russian then I would also expect that it would work on Greek, and when that isn't true it is odd. When that does not carry over to, say, Hebrew print/script character conversions and comparisons or hiragana/katakana conversions and comparisons I'm just disappointed because it seems the libs arbitrarily support lang X with special cases [X1, X2, X3, ...] but not some other. Hopefully I explained that adequately. -Craig From essen@REDACTED Tue Jan 31 15:14:51 2017 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 31 Jan 2017 15:14:51 +0100 Subject: [erlang-questions] Erlang zlib and concurrency Message-ID: <35054837-3b15-e44b-72d9-e89bfaafbb4c@ninenines.eu> Hello, The zlib module in Erlang can not do work concurrently, I believe. It always seem to use a single core. I have a vague memory of an Erlang Factory talk mentioning this and that they rewrote the algorithm in Erlang to speed things up. But I am seeing no such code around. Does anyone know of an open source implementation of zlib in Erlang? I personally need gzip only. If not, would anyone be willing to sponsor it? Shouldn't take more than a day or two. Cheers, -- Lo?c Hoguin https://ninenines.eu From g@REDACTED Tue Jan 31 16:16:29 2017 From: g@REDACTED (Garrett Smith) Date: Tue, 31 Jan 2017 09:16:29 -0600 Subject: [erlang-questions] Erlang cons function Message-ID: Is there a functional reference to the cons operator? Something equivalent to this: fun(H, T) -> [H|T] end I was hoping for erlang:'|'(H, T) but no dice. From eric.pailleau@REDACTED Tue Jan 31 18:48:25 2017 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Tue, 31 Jan 2017 18:48:25 +0100 Subject: [erlang-questions] Common Test logs In-Reply-To: <5b05e1d9-6cf5-8ad5-5321-22f06bdabe5d@ninenines.eu> References: <5b05e1d9-6cf5-8ad5-5321-22f06bdabe5d@ninenines.eu> Message-ID: <44b65f6d-8c27-ca66-e9de-817782392c81@wanadoo.fr> +1 I currently extends the erlang.mk clean target : clean:: -@REDACTED logs -type d -mtime +15 -delete but it is only a workaround. The timestamp part of directory name is hardcoded in ct_logs.erl . regards From carlsson.richard@REDACTED Tue Jan 31 19:26:10 2017 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 31 Jan 2017 19:26:10 +0100 Subject: [erlang-questions] Erlang cons function In-Reply-To: References: Message-ID: No, it's considered a language primitive, similar to fixed-size tuples {1,2}. That's not to say that there couldn't be a named function like the one you looked for, just that there isn't one because there has never been much of a need for it. /Richard 2017-01-31 16:16 GMT+01:00 Garrett Smith : > Is there a functional reference to the cons operator? Something > equivalent to this: > > fun(H, T) -> [H|T] end > > I was hoping for erlang:'|'(H, T) but no dice. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From silviu.cpp@REDACTED Tue Jan 31 20:27:56 2017 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Tue, 31 Jan 2017 21:27:56 +0200 Subject: [erlang-questions] Erlang zlib and concurrency In-Reply-To: <35054837-3b15-e44b-72d9-e89bfaafbb4c@ninenines.eu> References: <35054837-3b15-e44b-72d9-e89bfaafbb4c@ninenines.eu> Message-ID: Hi Loic, I found same problem while profiling the ejabberd server. I wrote a zlib implementation but unfortunately you can use it in case you want to do streaming only. As time the interface doesn't expose to many stuffs. We created it only to cover our bad performances in ejabberd. https://github.com/silviucpp/ezlib But also you can look at benchmark to have an idea how slow is zlib coming with erlang Silviu On Tue, Jan 31, 2017 at 4:14 PM, Lo?c Hoguin wrote: > Hello, > > The zlib module in Erlang can not do work concurrently, I believe. It > always seem to use a single core. I have a vague memory of an Erlang > Factory talk mentioning this and that they rewrote the algorithm in Erlang > to speed things up. But I am seeing no such code around. > > Does anyone know of an open source implementation of zlib in Erlang? I > personally need gzip only. > > If not, would anyone be willing to sponsor it? Shouldn't take more than a > day or two. > > Cheers, > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jan 31 22:34:52 2017 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 31 Jan 2017 21:34:52 +0000 Subject: [erlang-questions] Erlang cons function In-Reply-To: References: Message-ID: It is easy to define: cons(H,T) -> [H|T]. but you are perhaps better off by just using a fun-wrapper or writing it directly in places where you need it. Without support for currying much of the higher-order vocabulary seems somewhat weaker. On Tue, Jan 31, 2017 at 7:26 PM Richard Carlsson wrote: > No, it's considered a language primitive, similar to fixed-size tuples > {1,2}. That's not to say that there couldn't be a named function like the > one you looked for, just that there isn't one because there has never been > much of a need for it. > > /Richard > > 2017-01-31 16:16 GMT+01:00 Garrett Smith : > > Is there a functional reference to the cons operator? Something > equivalent to this: > > fun(H, T) -> [H|T] end > > I was hoping for erlang:'|'(H, T) but no dice. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: